Document evaluation apparatus and document evaluation method

Abstract

Streamline the evaluation of structured documents. 【Solution means】The document evaluation device (document processing device 100) creates an evaluation acquisition prompt that requests an evaluation result of evaluation target document information based on an evaluation criterion by referring to evaluation target document information including a correspondence between item identification information and an item value for one or more items, and an evaluation criterion related to the item. The document evaluation unit 113, an answer acquisition unit 118 that transmits the evaluation acquisition prompt to the language model server 210 and acquires an evaluation result directly or indirectly from the language model server 210. The document evaluation device includes an evaluation report unit 114 that creates a report acquisition prompt that requests a report in a predetermined format based on the evaluation result. The answer acquisition unit 118 transmits the report acquisition prompt to the language model server 210 and acquires a report directly or indirectly from the language model server 210.

Description

Technical Field

[0001] The present invention relates to a document evaluation apparatus and a document evaluation method for evaluating documents.

Background Art

[0002] Natural language processing using large language models is becoming widespread, and attention is being focused on automating and streamlining document processing such as evaluation / review that has hitherto been performed by humans using this technology. As an evaluation system related to language, there is a language transmission ability evaluation system described in Patent Document 1. This system assigns the role of an evaluator of a language ability evaluation test to a generative AI, and creates a prompt including an instruction to refer to the evaluation criteria of the language ability evaluation test and an instruction to confirm that the evaluation is based on the evaluation criteria. As a result, even when using a generative AI, evaluation results with little variation can be obtained.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Documents used in business have a structure such as chapters and items, and are evaluated / reviewed / checked according to criteria corresponding to the chapters and items. The evaluation criteria in the language transmission ability evaluation system described in Patent Document 1 are degree of achievement of tasks, vocabulary power, grammar ability, etc., and are not suitable for evaluating / reviewing / checking documents used in business.

[0005] The present invention has been made in view of such a background, and an object thereof is to provide a document evaluation apparatus and a document evaluation method for streamlining the evaluation of structured documents.

Means for Solving the Problems

[0006] To solve the above problems, a document evaluation apparatus according to the present invention includes: evaluation target document information including the correspondence between item identification information and item values for one or more items, and a document evaluation unit that creates an evaluation acquisition prompt for requesting an evaluation result of the evaluation target document information based on the evaluation criteria by referring to the evaluation criteria related to the items; a response acquisition unit that transmits the evaluation acquisition prompt to a language model server and acquires the evaluation result directly or indirectly from the language model server; and an evaluation report unit that creates a report acquisition prompt for requesting a report in a predetermined format based on the evaluation result. The response acquisition unit transmits the report acquisition prompt to the language model server and acquires the report directly or indirectly from the language model server. Moreover, the request for the evaluation result included in the evaluation acquisition prompt is a request in the form of a question for obtaining the evaluation result. The report acquisition prompt includes examples of questions included in the request in the form of a question, examples of the evaluation results as answers to the questions, examples of reports in the predetermined form for the examples of the questions and the examples of the answers, the request in the form of a question included in the evaluation acquisition prompt, and the evaluation results acquired by the answer acquisition unit.

Effect of the Invention

[0007] According to the present invention, it is possible to provide a document evaluation apparatus and a document evaluation method for streamlining the evaluation of structured documents. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Mode for Carrying Out the Invention

[0009] ≪Outline of Document Processing Apparatus≫ The following describes the overview of a document processing apparatus as a document analysis apparatus, a document evaluation apparatus, a document evaluation criterion extraction apparatus, and a document correction apparatus in a form (embodiment) for implementing the present invention. First, the document processing apparatus functions as a document analysis apparatus that extracts information considering the document structure from a tabular document such as a spreadsheet. More specifically, the document processing apparatus refers to the format information of the document, extracts the item value corresponding to the item identification information (item name), and outputs it as text.

[0010] Second, the document processing apparatus functions as a document evaluation apparatus that evaluates / reviews a document. More specifically, the document processing apparatus evaluates the document with reference to the text extracted from the document and the evaluation criteria of the document, and outputs a report (evaluation report). Third, the document processing apparatus functions as a document evaluation criterion extraction apparatus that extracts the evaluation criteria (check contents) of a document from a document checklist or the minutes of a review meeting. Fourth, the document processing apparatus functions as a document correction apparatus that corrects the fluctuations in the notation of a document. More specifically, the document processing apparatus detects synonyms from the document and corrects the fluctuations in the notation. After explaining the basic configuration of the document processing apparatus below, the functional configurations will be explained in the order of the document analysis apparatus, the document evaluation apparatus, the document evaluation criterion extraction apparatus, and the document correction apparatus.

[0011] ≪Configuration of Document Processing Apparatus≫ FIG. 1 is a functional block diagram of a document processing apparatus 100 according to the present embodiment. The document processing apparatus 100 is a computer and includes a control unit 110, a storage unit 130, and an input / output unit 180. User interface devices such as a display, a keyboard, and a mouse are connected to the input / output unit 180. The input / output unit 180 includes a communication device and can transmit and receive data to and from a language model server 210 and other devices. The language model server 210 is a server of a service (dialogue-type artificial intelligence service) that receives a prompt including an instruction and returns an answer to the prompt.

[0012] ≪Document Processing Apparatus: Storage Unit≫ The storage unit 130 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 130 stores a document database 131, a document format database 140, a document evaluation report rule database 150, an evaluation report database 160, and a program 138. Note that various stored contents of the storage unit 130 may be read as needed from those stored in an external storage device such as a cloud server.

[0013] The document database 131 stores documents to be processed by the document processing apparatus 100 in association with document identification information. The program 138 includes descriptions of processes executed by functional units provided in the control unit 110 described later. Other configurations of the storage unit 130 will be described later.

[0014] <<Document Processing Apparatus: Control Unit>> The control unit 110 is configured to include a CPU (Central Processing Unit), and includes a document analysis unit 111, a document format generation unit 112, a document evaluation unit 113, an evaluation report unit 114, an evaluation criterion extraction unit 115, a dictionary generation unit 116, a document correction unit 117, and a response acquisition unit 118. The response acquisition unit 118 sends a prompt generated by other functional units provided in the control unit 110 described later to the language model server 210 and receives a response directly or indirectly. The control unit 110 may be configured to include a GPU (Graphics Processing Unit), an NPU (Neural (network) Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like.

[0015] <<Document Processing Apparatus: Document Analysis Apparatus>> Hereinafter, the document processing apparatus 100 as a document analysis apparatus will be described. The document processing apparatus 100 refers to the format information (template information) of the tabular document, and extracts pairs of item identification information (item name) and item values for the items included in the tabular document.

[0016] ≪Document Analysis Apparatus: Tabular Document - Format Information≫ FIG. 2 is a diagram showing an example of the tabular document 410 according to the present embodiment. The tabular document 410 is a spreadsheet - type document, and is a document in which item values corresponding to item names are entered / described in the cells adjacent to the cells (columns) in which the item names are described. In FIG. 2, the item name is described in the cell immediately to the right of the cell in which the item name is described. For example, the item value corresponding to the item name "Required Delivery Date" is "2020 / 10 / 10". Note that the item value may also be described in the cell below the item name. Note that an example of the tabular document / spreadsheet - type document is an Excel (registered trademark) document, but it is not limited to this.

[0017] In a spreadsheet - type document, cells are identified by identification information indicating position information. For example, the identification information of the cell in which the item name "Required Delivery Date" is described is "B6:C7". The tabular document 410 is stored in the document database 131 in association with document identification information. The document database 131 may store text - type documents.

[0018] FIG. 3 is a diagram showing the format information document 420 (template document) of the tabular document according to the present embodiment. The cells of the item names are the same as those of the tabular document 410. In the cells where item values are to be entered, a character string with "#" added at the beginning of the item name (item identification information) is described. Hereinafter, unless otherwise particularly necessary to distinguish, the item identification information and the item identification information with "#" added at the beginning will be described without distinction. For example, the "Required Delivery Date" as item identification information and "#Required Delivery Date" are not distinguished.

[0019] ≪Document Analysis Apparatus: Document Format Database≫ In the document format database 140, the format information document 420 is stored in association with format information identification information. Also, the identification information (position information) and values of cells extracted from the format information document 420 by the document analysis unit 111 described later are stored in the document format database 140 in association with the format information document 420. In other words, the document format database 140 stores format information identification, the format information document 420 (see FIG. 3), and the correspondence between the identification information and values of the cells of the format information document 420 (see element 612 in FIG. 4 described later). Hereinafter, the correspondence between the identification information and values of the cells of the format information document 420 is also referred to as format information.

[0020] ≪Document Analysis Apparatus: Document Analysis Unit≫ Before starting the processing of the tabular document 410, the document analysis unit 111 extracts format information (the correspondence between the identification information and values of cells) from the format information document 420 and stores it in the document format database 140. The document analysis unit 111 extracts an item value corresponding to an item name (item identification information) from the tabular document 410 with reference to the format information. More specifically, the document analysis unit 111 first extracts document information (see element 613 in FIG. 4 described later), which is the correspondence between the identification information (position information) and values of cells, from the tabular document 410.

[0021] Next, the document analysis unit 111 generates an item value acquisition prompt 610 (see FIG. 4 described later) based on the extracted information. Subsequently, the document analysis unit 111 instructs the response acquisition unit 118 to send the item value acquisition prompt 610 to the language model server 210 and receives an item value acquisition result 430 (see FIG. 5 described later) as the response.

[0022] ≪Document Analysis Apparatus: Item Value Acquisition Prompt≫ FIG. 4 is a diagram showing the configuration of the item value acquisition prompt 610 according to the present embodiment. Generally, a prompt is composed of one or more elements. The item value acquisition prompt 610 includes elements 611 to 613. Element 611 indicates an instruction to extract an item value corresponding to item identification information starting with "#" from the document shown in element 613 with reference to the format (format information) shown in element 612.

[0023] Element 612 is the correspondence (format information) of the cell identification information and value extracted from the format information document 420 (see FIG. 3). Each row shows the correspondence of the identification information and the value. Element 613 is the correspondence (document information) of the cell identification information and value extracted from the tabular document 410 (see FIG. 2). Each row shows the correspondence of the identification information and the value.

[0024] ≪Document Analysis Device: Item Value Acquisition Result≫ FIG. 5 is a diagram showing the item value acquisition result 430 according to the present embodiment. The item value acquisition result 430 is data in JSON format specified by the element 611 of the item value acquisition prompt 610, and shows the correspondence between the item identification information (item name) and the item value. Note that JSON is an abbreviation for JavaScript Object Notation, which is the format of an object in the programming language JavaScript.

[0025] Taking the item identification information "#Business Order Certainty" as an example, the correspondence will be explained. The identification information of the cell in which the document format information "#Business Order Certainty" is described is "E2:F3" (see element 612 shown in FIGS. 3 and 4). The language model server 210 acquires "A" as the corresponding value in "E2:F3" in element 613 and includes it in the item value acquisition result 430. The document analysis unit 111 associates the item value acquisition result 430 with the tabular document 410 and stores it in the document database 131.

[0026] As described above, the document processing device 100 as a document analysis device includes a document analysis unit 111 that extracts document information (see element 613) indicating the correspondence between the identification information of each cell and the item value of the cell from the tabular document 410 including the cells in which the item values as items are entered.

[0027] The document analysis unit 111 creates an item value acquisition prompt 610 (see FIG. 4) that requests format information (see element 612) indicating the correspondence between the identification information of the cell in which the item value to be acquired is entered among the cells and the item identification information of the item, and an item value acquisition result 430 (see FIG. 5) indicating the correspondence between the item identification information and the item value entered in the cell corresponding to the item identification information, with reference to the document information. The document processing apparatus 100 as a document analysis apparatus includes a response acquisition unit 118 that transmits the item value acquisition prompt 610 to the language model server 210 and acquires the item value acquisition result 430 directly or indirectly from the language model server 210.

[0028] ≪Document Analysis Apparatus: Document Analysis Processing≫ FIG. 6 is a flowchart of the document analysis processing according to the present embodiment. In step S11, the document analysis unit 111 extracts the identification information and values of the cells from the tabular document 410 that is the document to be processed (see element 613 shown in FIG. 4).

[0029] Step S12 is a document format acquisition process of acquiring the format information corresponding to the tabular document 410 of the document to be processed from the document format database 140. Details will be described later with reference to FIG. 7. In step S13, the document analysis unit 111 generates an item value acquisition prompt 610 (see FIG. 4). Here, element 612 is the format information acquired in step S12. Element 613 is the correspondence between the identification information and values of the cells acquired in step S11.

[0030] In step S14, the document analysis unit 111 instructs the response acquisition unit 118 to send the item value acquisition prompt 610 to the language model server 210. Next, the document analysis unit 111 associates the item value acquisition result 430, which is the response (see FIG. 5), with the tabular document 410 and stores it in the document database 131.

[0031] ≪Document Analysis Apparatus: Document Format Acquisition Processing≫ FIG. 7 is a flowchart of the document format acquisition process according to the present embodiment. The document format acquisition process in step S12 (see FIG. 6) will be described with reference to FIG. 7. In step S21, the document analysis unit 111 starts a process of repeating the process of step S22 for each format information stored in the document format database 140. Note that the format information is the correspondence between the identification information of the cells of the format information document 420 and the values (see element 612 in FIG. 4).

[0032] In step S22, the document analysis unit 111 calculates the degree of match between the format information and the tabular document 410 that is the processing target of the document analysis process (see FIG. 6). The degree of match is the ratio or number that matches the values and identification information of the cells included in the tabular document 410 for the values and identification information of the cells in the format information that do not start with "#". In other words, it is the ratio or number that matches the cells with the same item names in the tabular document 410 among the cells in the format information document 420 where the item names that do not start with "#" are described. Note that the cells of the item names are, for example, the cells of "(Sales) Order Accuracy" in "B2:D3" and "Completion" in "F6:F7". In step S23, the document analysis unit 111 sets the format information with the maximum degree of match calculated in step S22 as the format information of the tabular document 410 of the processing target document.

[0033] As described above, the identification information of the cell is information indicating the position of the cell in the tabular document 410. The document analysis unit 111 refers to the format information in the document format database 140 that stores one or more pieces of format information, and creates the item value acquisition prompt 610 by referring to the format information in which the number of cells with matching cell identification information and cell values is the largest between the cells included in the format information and the cells included in the document information (see element 613).

[0034] ≪Features of the Document Analysis Apparatus≫ The document processing apparatus 100 as a document analysis apparatus extracts an item value acquisition result 430 (see FIG. 5) from a tabular document 410 (see FIG. 2) with reference to format information (see element 612 described in FIG. 4). Even for a document with a complex tabular format, it is possible to obtain an item value acquisition result that is the correspondence between item identification information and item values necessary for performing document processing such as review / evaluation. According to the document format acquisition process (see FIG. 7), format information that most closely matches the format of the document to be processed is acquired. There is no need to specify a format (template document) for each document, and the item value acquisition result can be efficiently obtained.

[0035] ≪Modification example of document analysis apparatus: Item value acquisition result≫ The item value acquisition result 430 (see FIG. 5) was in JSON format, but it is not limited to this. It may be in text format. FIG. 8 is a diagram showing a text format item value acquisition result 430A according to a modification example of the present embodiment. The item value acquisition result 430A is text format data with the same content as the item value acquisition result 430.

[0036] ≪Document analysis apparatus: Document format generation unit≫ In the above-described embodiment, the format information (see element 612 described in FIG. 4) is extracted from a format information document 420 (see FIG. 3) in which item identification information starting with "#" is described. The document format generation unit 112 extracts format information from a format information document 420A (see FIG. 9 to be described later) in which the item identification information is blank. FIG. 9 is a diagram showing a format information document 420A (template document) in which the item identification information is blank according to a modification example of the present embodiment. Compared with the format information document 420, the item identification information is blank.

[0037] FIG. 10 is a flowchart of the format information acquisition process according to the present embodiment. With reference to FIG. 10, the process of acquiring format information from a format information document 420A in which the entry fields (cells) for item values are blank will be described. By the document format acquisition process, it becomes possible to acquire format information without preparing a format information document 420 in which item identification information starting with "#" is entered.

[0038] In step S31, the document format generation unit 112 extracts the correspondence between the cell identification information and the value (template document information) from the format information document 420A. In step S32, the document format generation unit 112 acquires cells that are blank cells, where the cell above is not blank and the cell to the left is blank. Note that the cells above / below / to the left / to the right of a cell can be determined from the identification information indicating the position. Also, assume that the cell above has the same width and the cell to the left has the same height.

[0039] In step S33, the document format generation unit 112 starts a process of repeating the process of step S34 for each cell acquired in step S32. In step S34, the document format generation unit 112 uses the item identification information obtained by adding "#" to the beginning of the value (string) of the cell above and the cell identification information as format information.

[0040] In step S35, the document format generation unit 112 acquires cells that are blank cells, where the cell to the left is not blank and the cell above is blank. Note that the cell above has the same width and the cell to the left has the same height. In step S36, the document format generation unit 112 starts a process of repeating the process of step S37 for each cell acquired in step S35.

[0041] In step S37, the document format generation unit 112 uses the item identification information obtained by adding "#" to the beginning of the value (string) of the cell to the left and the cell identification information as format information. In step S38, the document format generation unit 112 stores the format information obtained in steps S34 and S37 in the document format database 140 as the format information of the format information document 420A.

[0042] As described above, the document processing apparatus 100 as a document analysis apparatus includes a document format generation unit 112 that extracts template document information indicating the correspondence between the identification information indicating the position of each cell and the value entered in the cell from a template document (see the format information document 420A shown in FIG. 9) with unentered item values, which serves as a template for the tabular document 410.

[0043] When the cell above the blank cell, which is a blank cell, is not blank and the cell to the left is blank, the document format generation unit 112 associates the identification information of the blank cell with the value of the cell above as item identification information. Also, when the cell to the left of the blank cell is not blank and the cell above is blank, the document format generation unit 112 associates the identification information of the blank cell with the value of the cell to the left as item identification information. The document format generation unit 112 uses the correspondence between the identification information of the associated blank cell and the item identification information as format information.

[0044] ≪Document Processing Apparatus: Document Evaluation Apparatus≫ Hereinafter, the document processing apparatus 100 as a document evaluation apparatus will be described. The document processing apparatus 100 performs evaluation / review on a document including items, such as the tabular document 410, according to the evaluation criteria for each item and reports the results. Even for a document in text format that includes structures such as chapters, sections, and items, it is also a processing target of the document evaluation apparatus. In the following description, elements / components of the document structure such as chapters and sections are also referred to as items.

[0045] ≪Document Evaluation Apparatus: Outline of Processing≫ FIG. 11 is a diagram for explaining the outline of document evaluation report processing (see FIG. 24 described later) by the document processing apparatus 100 according to the present embodiment. The document evaluation unit 113 generates an evaluation acquisition prompt 620 (see FIG. 12 described later) based on the evaluation criteria stored in the document evaluation report rule database 150 and the item value acquisition result 430 (see FIG. 5). The evaluation acquisition prompt 620 is transmitted to the language model server 210, and an evaluation result 450 (see FIG. 13 described later) is returned. The evaluation result 450 is the result of the evaluation / review of the item value acquisition result 430 according to the evaluation criteria.

[0046] The evaluation report unit 114 generates a report acquisition prompt 630 (see FIG. 15 described later) based on the evaluation result 450 and the report examples stored in the document evaluation report rule database 150. The report acquisition prompt 630 is sent to the language model server 210, and a report 460 (see FIG. 17 described later) is returned. Hereinafter, the details of the document processing apparatus 100 as a document evaluation apparatus will be described.

[0047] ≪Document Evaluation Apparatus: Document Evaluation Report Rule Database≫ In the document evaluation report rule database 150, the evaluation criteria for evaluating / reviewing a document are stored in association with format information (see the element 612 shown in FIG. 4 of the document format database 140). In the format information, not only the items (item identification information and item values) of the tabular document, but also the components such as chapters, sections, charts, etc. included in the text document are regarded as items. The evaluation criteria (see the element 622 in FIG. 12 described later) are the evaluation criteria for each of these items. Since the format information includes one or more items, one or more evaluation criteria are associated with one piece of format information.

[0048] In addition, the document evaluation report rule database 150 also stores questions corresponding to the evaluation criteria (see the element 624 in FIG. 12 described later). The questions are for obtaining the evaluation results for each item or evaluation criterion. Since the format information includes one or more items, one or more questions are associated with one piece of format information.

[0049] Furthermore, the document evaluation report rule database 150 also stores report examples (see the elements 632 to 635 in FIG. 15 described later) for the evaluation results obtained by evaluating / reviewing based on the evaluation criteria. The report examples are the report examples for the evaluation results for each item. Since the format information includes one or more items, one or more report examples are associated with one piece of format information. In addition, the document evaluation report rule database 150 may store a document (also referred to as a checklist) that describes the viewpoints, check contents, and check rules for evaluating / reviewing a document.

[0050] <<Document Evaluation Device: Evaluation Report Database>> In the evaluation report database 160, the evaluation results of evaluating documents and the reports summarizing the evaluation results are stored in association with the document identification information of the documents to be evaluated. Further, in the evaluation report database 160, the minutes of the meetings at which the documents were evaluated / reviewed while referring to the documents and reports are stored in association with the document identification information of the documents to be evaluated.

[0051] <<Document Evaluation Device: Document Evaluation Unit>> The document evaluation unit 113 evaluates / reviews a document by referring to the evaluation criteria related to the items constituting the document. The document evaluation unit 113 generates an evaluation acquisition prompt 620 (see FIG. 12 to be described later) based on the evaluation criteria and document information. Subsequently, the document evaluation unit 113 instructs the response acquisition unit 118 to send the evaluation acquisition prompt 620 to the language model server 210 and receives the evaluation result (simply referred to as evaluation) as the response.

[0052] <<Document Evaluation Device: Evaluation Acquisition Prompt>> FIG. 12 is a diagram showing the configuration of the evaluation acquisition prompt 620 according to the present embodiment. The evaluation acquisition prompt 620 includes elements 621 to 624. Element 621 indicates an instruction / direction to answer the question shown in element 624 for the document that is the evaluation target shown in element 623.

[0053] Element 622 is the evaluation criteria for the items included in the document that is the evaluation target (see element 623). The evaluation criteria are stored in the document evaluation report rule database 150. Element 623 is the document information (evaluation target document information) of the evaluation target. The document information at element 623 in FIG. 12 is the correspondence between the item identification information and the item values acquired by the document analysis unit 111 from the tabular document (see the item value acquisition result 430 shown in FIG. 5). The document information of element 623 may be text format document information created by a word processor. For example, it may be the text format item value acquisition result 430A (see FIG. 8).

[0054] Element 624 is a question for obtaining an evaluation result according to the evaluation criteria in element 622. The question may include procedures for obtaining the evaluation result. The questions described in FIG. 12 include: (1) correctly counting the numbers of "high", "medium", and "low" in "evaluation result"; (2) obtaining a project risk rank according to the evaluation criteria; and (3) comparing the obtained project risk rank with the project risk rank included in the evaluation target. Thus, the evaluation acquisition prompt 620 is a prompt for obtaining an evaluation result with reference to the evaluation criteria related to the item of "project risk rank".

[0055] ≪Document Evaluation Device: Evaluation Result≫ FIG. 13 is a diagram showing the evaluation result 450 according to the present embodiment. FIG. 14 is a diagram showing the evaluation result 450A according to the present embodiment. The evaluation results 450 and 450A are answers along the procedures shown in the questions of element 624 included in the evaluation acquisition prompt 620. Note that the evaluation results 450 and 450A have different evaluation results depending on whether they match or not in the comparison of the project risk rank, which is the last procedure. As described above, the document evaluation unit 113 uses the evaluation acquisition prompt 620 to inquire of the language model server 210 for each evaluation criterion of each item included in the format information corresponding to the document to be evaluated, and obtains the evaluation results 450 and 450A.

[0056] As described above, the document processing apparatus 100 as a document evaluation apparatus includes a document evaluation unit 113 that creates an evaluation acquisition prompt 620 for requesting evaluation results 450 and 450A of the document information to be evaluated based on the evaluation criteria with reference to the evaluation target document information (refer to element 623) including the correspondence between the item identification information and the item value for one or more items, and the evaluation criteria related to the items (refer to element 622). The document processing apparatus 100 as a document evaluation apparatus includes a response acquisition unit 118 that transmits the evaluation acquisition prompt 620 to the language model server 210 and directly or indirectly obtains the evaluation results 450 and 450A from the language model server 210. The request for the evaluation result included in the evaluation acquisition prompt 620 is a question-form request (see element 624) for the evaluation results 450, 450A.

[0057] <<Document Evaluation Device: Evaluation Report Section>> The evaluation report section 114 generates an evaluation report with reference to the evaluation results 450, 450A. Specifically, the evaluation report section 114 generates a report acquisition prompt 630 (see FIG. 15 to be described later) based on the evaluation results and report examples. Subsequently, the evaluation report section 114 instructs the response acquisition section 118 to send the report acquisition prompt 630 to the language model server 210 and receives the report as the response. Similar to the document evaluation section 113, the evaluation report section 114 generates a report acquisition prompt 630 for each item constituting the document and receives a report 460 (see FIG. 17 to be described later).

[0058] <<Document Evaluation Device: Report Acquisition Prompt>> FIGS. 15 and 16 are diagrams showing the configuration of the report acquisition prompt 630 according to the present embodiment. The report acquisition prompt 630 includes elements 631 to 637. Element 631 indicates an instruction to report on the question shown in element 636 and the answer (evaluation result) to the question shown in element 637 with reference to the report examples shown in elements 632 to 635.

[0059] Element 632 is an example of a question and an answer where the determination results do not match. Element 633 is an example of a report when the determination results do not match as shown in element 632. The report includes the presence or absence of pointed-out matters, the pointed-out content, and amendments. Element 634 is an example of a question and an answer where the determination results match. Element 635 is an example of a report when the determination results match as shown in element 634. The report includes the presence or absence of pointed-out matters, the pointed-out content, and amendments, but the content is equal to none.

[0060] Moving on to FIG. 16, the description of the report acquisition prompt 630 continues. Element 636 is a question for obtaining the evaluation result in element 624 of the evaluation acquisition prompt 620 (see FIG. 12). Element 637 is the evaluation result 450 (see FIG. 13), which is an answer to the question.

[0061] ≪Document Evaluation Device: Report≫ FIG. 17 is a diagram showing the report 460 according to the present embodiment. The answer, which is element 637, is the evaluation result 450 where the determination results do not match, and corresponds to the question and answer in element 632. For this reason, the report 460 is a report along the report in element 633. As described above, the evaluation report unit 114 uses the report acquisition prompt 630 including report examples corresponding to the evaluation results 450, 450A for each item acquired by the document evaluation unit 113 to inquire of the language model server 210 and obtains the report 460.

[0062] As described above, the document processing device 100 as a document evaluation device includes an evaluation report unit 114 that creates a report acquisition prompt 630 for requesting a report 460 in a predetermined format based on the evaluation results 450, 450A. The answer acquisition unit 118 transmits the report acquisition prompt 630 to the language model server 210 and acquires the report 460 directly or indirectly from the language model server 210.

[0063] The report acquisition prompt 630 includes examples of questions included in the question-form request (see elements 632, 634), examples of evaluation results as answers to the questions (see elements 632, 634), examples of reports in a predetermined format for the examples of the questions and the examples of the answers (see elements 633, 635), the question-form request included in the evaluation acquisition prompt 620 (see element 636), and the evaluation results acquired by the answer acquisition unit 118 (see element 637).

[0064] The evaluation target document information (see element 623 described in FIG. 12) includes an evaluation item (for example, "project risk rank"), which is an item whose item value is determined according to the number of items whose item values are within a predetermined range (for example, "high", "medium", "low") where the item values are one or more predetermined values. The evaluation criteria (see element 622) include the criteria for determining the values of the evaluation items. The evaluation acquisition prompt 620 includes a request to count the number of items whose item values are within a predetermined range, a request to obtain a counted evaluation item value that is the value of the evaluation item determined according to the counted result, a counted evaluation item value, and a request to report whether the counted evaluation item value matches the described evaluation item value that is the value of the evaluation item included in the document information to be evaluated (see element 624).

[0065] The predetermined form of the report (see element 633 described in FIG. 15) is a form for reporting a discrepancy when the counted evaluation item value and the described evaluation item value do not match. The report in the predetermined form (see element 633) is a form that includes an amendment to correct the described evaluation item value to the counted evaluation item value when the counted evaluation item value and the described evaluation item value do not match.

[0066] <<Variants of the evaluation acquisition prompt and the report acquisition prompt>> The evaluation criteria (see element 622) included in the evaluation acquisition prompt 620 (see FIG. 12) are evaluation criteria in which another item value ("AA", "A", "B", "C") is determined according to the number of item values ("high", "medium"). Some examples of evaluation criteria / check contents other than such a format are shown.

[0067] FIG. 18 is a diagram showing an evaluation acquisition prompt 620A according to the present embodiment. The evaluation criteria in element 622A of the evaluation acquisition prompt 620A are evaluation criteria in which another item value ("final project risk rank") is determined based on two item values ("project risk rank according to scale", "project risk rank according to required quality"). In other words, the evaluation acquisition prompt 620A is a prompt for obtaining an evaluation result related to the item of "final project risk rank".

[0068] FIG. 19 is a diagram showing a report acquisition prompt 630A according to the present embodiment. The report acquisition prompt 630A corresponds to the evaluation acquisition prompt 620A and is a prompt for acquiring a report on the evaluation result related to the item of "final project risk rank". Element 632A is an example where the determination results do not match. Element 633A is an example of a report when the determination results do not match as shown in element 632A. Element 634A is an example where the determination results match. Element 635A is an example of a report when the determination results match as shown in element 634A.

[0069] As described above, the evaluation target document information (see element 623A) includes an evaluation item (for example, "final project risk rank") which is an item whose item value is determined according to the maximum or minimum item value among a plurality of comparison target item values (for example, "project risk rank by scale", "project risk rank by required quality").

[0070] The evaluation criteria include the criteria for determining the value of the evaluation item. The evaluation acquisition prompt 620A includes a request for obtaining a comparison evaluation item value which is the value of the evaluation item determined according to the result of comparing the comparison target item values, and a request for reporting whether the comparison evaluation item value matches the described evaluation item value which is the value of the evaluation item included in the evaluation target document information (see element 624A).

[0071] The predetermined form of the report (see element 633A) is a form for reporting a discrepancy when the comparison evaluation item value and the described evaluation item value do not match. The report in the predetermined form (see element 633A) is a form including an amendment for correcting the described evaluation item value to the comparison evaluation item value when the comparison evaluation item value and the described evaluation item value do not match.

[0072] FIG. 20 is a diagram showing an evaluation acquisition prompt 620B according to the present embodiment. The evaluation criteria in element 622B of the evaluation acquisition prompt 620B are criteria for determining whether the item value is a predetermined value (for example, blank) and determining the appropriateness (presence or absence of violation) of the item value. FIG. 21 is a diagram showing a report acquisition prompt 630B according to the present embodiment. Element 632B is an example of a violation. Element 633B is an example of a report when there is a violation as shown in element 632B. Element 634B is an example of no violation. Element 635B is an example of a report when there is no violation as shown in element 634B.

[0073] As described above, the evaluation criteria (element 622B) include prohibited item identification information which is item identification information for which it is prohibited that the item value is a predetermined value (for example, blank). The evaluation acquisition prompt 620B includes a request to acquire the item value of the prohibited item identification information included in the document information to be evaluated, and a request to obtain prohibited value entry item identification information indicating the item identification information for which the acquired item value is a predetermined value. The predetermined form of the report (see element 633B) is a form including prohibited value entry item identification information.

[0074] FIG. 22 is a diagram showing an evaluation acquisition prompt 620C according to the present embodiment. The evaluation criteria in element 622C of the evaluation acquisition prompt 620C are evaluation criteria for determining the appropriateness of the item value. FIG. 23 is a diagram showing a report acquisition prompt 630C according to the present embodiment. Element 632C is an example where the item value is inappropriate (violates the evaluation criteria). Element 633C is an example of a report when the item value is inappropriate as shown in element 632B. Element 634C is an example where the item value is appropriate. Element 635C is an example of a report when the item value is appropriate as shown in element 634B.

[0075] <<Document Evaluation Device: Document Evaluation Report Processing>> FIG. 24 is a flowchart of the document evaluation report processing according to this embodiment. The process of obtaining the evaluation result for a document and further generating a report will be described with reference to FIG. 24.

[0076] In step S41, the document evaluation unit 113 acquires document information that is the target of the evaluation report. For example, for a tabular document, the document evaluation unit 113 acquires the item value acquisition result 430 (see FIG. 5) extracted by the document analysis unit 111 from the document database 131. Next, the document evaluation unit 113 acquires evaluation criteria, questions, and report examples from the document evaluation report rule database 150 based on the format information corresponding to the document.

[0077] In step S42, the document evaluation unit 113 starts a process of repeating steps S43 to S44 for each item having an evaluation criterion. In step S43, the document evaluation unit 113 generates an evaluation acquisition prompt 620 (see FIG. 12). Here, the element 622 is the evaluation criterion corresponding to the item among the evaluation criteria acquired in step S41. Also, the element 624 is the question corresponding to the evaluation criterion. The element 623 is the document information acquired in step S41. The element 624 is the question corresponding to the evaluation criterion (corresponding to the item) among the questions acquired in step S41.

[0078] In step S44, the document evaluation unit 113 instructs the answer acquisition unit 118 to send the evaluation acquisition prompt 620 to the language model server 210. Next, the document evaluation unit 113 stores the evaluation results 450, 450A (see FIGS. 13 and 14), which are the answers, in the evaluation report database 160 in association with the document identification information of the document.

[0079] In step S45, the evaluation report unit 114 starts a process of repeating steps S46 to S47 for each item having an evaluation criterion. In step S46, the evaluation report unit 114 generates a report acquisition prompt 630 (see FIGS. 15 to 16). Here, elements 632 to 635 are report examples corresponding to items (evaluation criteria) among the report examples acquired in step S41. Further, element 636 is a question corresponding to the item, and is the question in element 624 of the evaluation acquisition prompt 620 generated in step S43. Element 637 is an evaluation result which is the answer acquired in step S44.

[0080] In step S47, the evaluation report unit 114 instructs the answer acquisition unit 118 to send the report acquisition prompt 630 to the language model server 210. Next, the evaluation report unit 114 stores the report 460 (see FIG. 17), which is the answer, in the evaluation report database 160 in association with the document identification information of the document. In step S48, the evaluation report unit 114 merges the reports acquired in step S47 to create a report on the evaluation of the document, and stores it in the evaluation report database 160.

[0081] ≪Features of the document evaluation apparatus≫ The document processing apparatus 100 as a document evaluation apparatus refers to the evaluation criteria (see elements 622, 622A, 622B, 622C described in FIGS. 12, 18, 20, 22) and acquires evaluation results 450, 450A for each item. Further, the document processing apparatus 100 refers to the report examples (see elements 632 to 635, 632A to 635A, 632B to 635B, 632C to 635C described in FIGS. 15, 19, 21, 23) and acquires reports 460 for each item. By evaluating / reviewing each item included in the document and generating a report, the document processing apparatus 100 can acquire highly accurate evaluation results and reports corresponding to the item. Also, by preparing evaluation criteria and questions corresponding to the item, it becomes possible to expect to obtain even more accurate evaluation results and reports.

[0082] ≪Document processing apparatus: Document evaluation criterion extraction apparatus≫ Hereinafter, the document processing apparatus 100 as a document evaluation criterion extraction apparatus will be described. The document processing apparatus 100 collects evaluation criteria, check contents, and check rules for each item for a document including items, such as the tabular document 410. The document processing apparatus 100 extracts and collects from the minutes of the evaluation / review meeting and the document describing the evaluation criteria and check contents. Hereinafter, the evaluation criteria, check contents, and check rules are collectively referred to as evaluation criteria.

[0083] ≪Document Evaluation Criterion Extraction Apparatus: Evaluation Criterion Extraction Unit≫ FIG. 25 is an example of a document 510 including evaluation criteria according to the present embodiment. The document 510 is, for example, the minutes of a meeting for evaluating / reviewing a document. The minutes include evaluation criteria including review viewpoints and rules. The evaluation criterion extraction unit 115 extracts evaluation criteria from the document including the evaluation criteria. More specifically, the evaluation criterion extraction unit 115 generates an evaluation criterion acquisition prompt 640 (see FIG. 26 described later) that instructs / orders the extraction of evaluation criteria from the document. Subsequently, the evaluation criterion extraction unit 115 instructs the response acquisition unit 118 to send the evaluation criterion acquisition prompt 640 to the language model server 210 and receives the evaluation criteria as a response. The evaluation criterion extraction unit 115 stores the received evaluation criteria in the document evaluation report rule database 150 in association with the format information of the document to be evaluated / reviewed.

[0084] ≪Document Evaluation Criterion Extraction Apparatus: Evaluation Criterion Acquisition Prompt≫ FIG. 26 is a diagram showing the configuration of the evaluation criterion acquisition prompt 640 according to the present embodiment. The evaluation criterion acquisition prompt 640 includes elements 641 to 642. Element 641 indicates an instruction / order to generate and acquire evaluation criteria with reference to the actual review points shown in element 642. Element 642 includes the text information included in the document 510.

[0085] ≪Document Evaluation Criterion Extraction Apparatus: Evaluation Criteria≫ FIG. 27 is a diagram showing the acquired evaluation criteria 520 according to the present embodiment. The evaluation criteria 520 are general evaluation criteria excluding the specific content of individual documents from the content of the review points included in the element 642. In this way, the evaluation criteria extraction unit 115 extracts and acquires the evaluation criteria from the documents including the evaluation target.

[0086] As described above, the document processing apparatus 100 as a document evaluation criteria extraction apparatus includes an evaluation criteria extraction unit 115 that creates an evaluation criteria acquisition prompt 640 (see FIG. 26) for extracting the evaluation criteria of the document by referring to the minutes including the review results of the document. Further, the document processing apparatus 100 as a document evaluation criteria extraction apparatus includes a response acquisition unit 118 that transmits the evaluation criteria acquisition prompt 640 to the language model server 210 and acquires the evaluation criteria directly or indirectly from the language model server 210. The evaluation criteria extraction unit 115 stores the evaluation criteria in the storage unit 130 (see the document evaluation report rule database 150).

[0087] ≪Document Evaluation Criteria Extraction Apparatus: Variation Example of Evaluation Criteria Acquisition Prompt≫ The evaluation criteria acquired using the evaluation criteria acquisition prompt 640 (see FIG. 26) are evaluation criteria related to the document that is the target of review / evaluation, and by extension, documents in the same format as the said document. The evaluation criteria extraction unit 115 may acquire the evaluation criteria for each item determined by the document format (format information).

[0088] FIG. 28 is a diagram showing the evaluation criteria acquisition prompt 640A according to the present embodiment. The evaluation criteria acquisition prompt 640A includes an element 643A that is not present in the evaluation criteria acquisition prompt 640. The element 641A indicates an instruction / command to generate and acquire the evaluation criteria for each item of the document in the element 643A. FIG. 29 shows the evaluation criteria 520A when the evaluation criteria acquisition prompt 640A according to the present embodiment is used. The evaluation criteria are extracted for each item (chapter).

[0089] As described above, the evaluation criterion extraction unit 115 creates an evaluation criterion acquisition prompt 640A that extracts evaluation criteria for each item (refer to element 643A described in FIG. 28) of the document by referring to the items described in the document and the minutes of the meeting including the review results of the document. The evaluation criterion extraction unit 115 stores the evaluation criteria in the storage unit 130 (refer to the document evaluation report rule database 150) as evaluation criteria related to the items.

[0090] The document to be the extraction source is not necessarily a text document such as minutes of a meeting. Text information may be extracted from a checklist document in tabular form, and the evaluation criteria that are the contents to be checked may be extracted. FIG. 30 is a diagram showing an evaluation criterion acquisition prompt 640B (check content acquisition prompt) according to the present embodiment. The evaluation criterion acquisition prompt 640B includes elements 641B and 642B.

[0091] Element 641B indicates an instruction / direction to generate and acquire check contents (evaluation criteria) for each item (chapter) from the text information in element 642B. Element 642B indicates text information extracted from a checklist document ("checklist for functional specifications" described in FIG. 30). FIG. 31 shows evaluation criteria 520B (check contents) when the evaluation criterion acquisition prompt 640B according to the present embodiment is used. Evaluation criteria are extracted collectively for each item (chapter).

[0092] As described above, the evaluation criterion extraction unit 115 creates a check content acquisition prompt (refer to the evaluation criterion acquisition prompt 640B) that extracts check contents for each item (for example, chapter) from a checklist document including the check contents for each item of the document. The response acquisition unit 118 transmits the check content acquisition prompt to the language model server 210 and acquires the check content directly or indirectly from the language model server 210. The evaluation criterion extraction unit 115 stores the check content in the storage unit 130 (the document evaluation report rule database 150) as evaluation criteria related to the items.

[0093] <<Document Evaluation Criterion Extraction Device: Evaluation Criterion Extraction Process>> FIG. 32 is a flowchart of the evaluation criterion extraction process according to this embodiment. In step S51, the evaluation criterion extraction unit 115 acquires the minutes of the meeting in which the document was evaluated / reviewed from the evaluation report database 160. In step S52, the evaluation criterion extraction unit 115 starts a process of repeating steps S53 to S54 for each set of minutes.

[0094] In step S53, the evaluation criterion extraction unit 115 generates an evaluation criterion acquisition prompt 640A (see FIG. 28). Here, element 642A is the text information included in the minutes. Element 643A is an item (item identification information) included in the format information of the document to be evaluated / reviewed in the minutes.

[0095] In step S54, the evaluation criterion extraction unit 115 instructs the response acquisition unit 118 to send the evaluation criterion acquisition prompt 640A to the language model server 210. Next, the evaluation criterion extraction unit 115 stores the evaluation criterion 520A (see FIG. 29), which is the response, in the document evaluation report rule database 150 in association with the item of the format information. Note that the evaluation criterion extraction unit 115 may inquire the user of the document processing device 100 as to whether or not to adopt the evaluation criterion before storing the evaluation criterion 520A in the document evaluation report rule database 150. Steps S55 to S58 are the same processes as steps S51 to S54, and the process is performed for the checklist instead of the minutes.

[0096] <<Features of the Evaluation Criterion Extraction Device>> The document processing device 100 as an evaluation criterion extraction device extracts evaluation criteria for each document format (format information) and for each item constituting the document from the minutes or the checklist. It becomes possible to acquire evaluation criteria without using manual labor. By collecting evaluation criteria and manually organizing them as necessary and referring to them in the document evaluation report process, it can be expected that the quality of document evaluation by the document processing device 100 will be improved.

[0097] <<Document Processing Device: Document Correction Device>> Hereinafter, the document processing device 100 as a document correction device will be described. The document processing device 100 performs document correction including eliminating fluctuations in word notation corresponding to documents and the tasks related to the documents.

[0098] <<Document Correction Device: Outline of Processing>> FIG. 33 is a diagram for explaining the outline of document correction processing (see FIG. 40 described later) by the document processing device 100 according to the present embodiment. The dictionary generation unit 116 generates a synonym acquisition prompt 650 (see FIG. 34 described later) that refers to one or more related documents 540 such as documents related to the same task. The synonym acquisition prompt 650 is transmitted to the language model server 210, and a synonym dictionary 550 (see FIG. 35 described later) is returned. The synonym dictionary 550 is a synonym dictionary specialized for the context included in the document 540 and the task related to the document 540.

[0099] The document correction unit 117 generates a corrected document acquisition prompt 660 (see FIG. 36 described later) that refers to the synonym dictionary 550. The corrected document acquisition prompt 660 is transmitted to the language model server 210, and the corrected document 560 is returned. Hereinafter, the details of the document processing device 100 as a document correction device will be described.

[0100] <<Document Correction Device: Dictionary Generation Unit>> The dictionary generation unit 116 generates a synonym acquisition prompt 650 (see FIG. 34 described later) that instructs / orders the generation of a synonym dictionary. Subsequently, the dictionary generation unit 116 instructs the answer acquisition unit 118 to send the synonym acquisition prompt 650 to the language model server 210 and receives the synonym dictionary 550 (see FIG. 35 described later) as the answer.

[0101] <<Document Correction Device: Synonym Acquisition Prompt>> FIG. 34 is a diagram showing the configuration of the synonym acquisition prompt 650 according to the present embodiment. The synonym acquisition prompt 650 includes elements 651 to 652. Element 651 indicates an instruction to obtain a thesaurus (a group of synonyms) by referring to the text information of document 540 shown in element 652. The instruction includes criteria for selecting which word as the headword among the synonyms. Examples of the criteria include katakana notation priority, kanji notation priority, and abbreviation priority. Element 652 includes the text information contained in document 540.

[0102] ≪Document correction device: Thesaurus≫ FIG. 35 is a diagram showing the obtained thesaurus 550 according to the present embodiment. The thesaurus 550 uses a word that conforms to the headword selection criteria instructed to element 651 as the headword (a key in JSON format).

[0103] ≪Document correction device: Document correction unit≫ The document correction unit 117 generates a corrected document acquisition prompt 660 (see FIG. 36 described later) that instructs the correction of a document. Subsequently, the document correction unit 117 instructs the response acquisition unit 118 to send the corrected document acquisition prompt 660 to the language model server 210 and receives the corrected document as the response. The document correction unit 117 stores the corrected document in the document database 131 in association with the document to be configured.

[0104] ≪Document correction device: Corrected document acquisition prompt≫ FIG. 36 is a diagram showing the configuration of the corrected document acquisition prompt 660 according to the present embodiment. The corrected document acquisition prompt 660 includes elements 661 to 663. Element 661 indicates an instruction to correct and proofread the document shown in element 663 by referring to the thesaurus 550 shown in element 662. Element 661 includes instructions to correct not only the fluctuations of words but also the fluctuations of tenses and to correct the end of a sentence to the "desu·masu style". The thesaurus 550 is shown in element 662. The text information of the document to be corrected is shown in element 663.

[0105] As described above, the document processing apparatus 100 as a document correction apparatus includes a dictionary generation unit 116 that creates a synonym acquisition prompt 650 that requests a synonym group in which words included in the text information of a document group (see element 652 described in FIG. 34) including one or more documents are grouped by synonyms. The document processing apparatus 100 as a document correction apparatus includes a response acquisition unit 118 that transmits the synonym acquisition prompt 650 to the language model server 210 and acquires a synonym group (see the synonym dictionary 550 described in FIG. 35) directly or indirectly from the language model server 210.

[0106] The document processing apparatus 100 as a document correction apparatus includes a document correction unit 117 that creates a corrected document acquisition prompt 660 (see FIG. 36) that requests a corrected document in which the variation in the word representation of the documents included in the document group is corrected with reference to the synonym group. The response acquisition unit 118 transmits the corrected document acquisition prompt 660 to the language model server 210 and acquires a corrected document directly or indirectly from the language model server 210.

[0107] The synonym acquisition prompt 650 includes a request to use, as a headword, a word that satisfies a predetermined criterion for each synonym group (see element 651). The predetermined criterion is any one of katakana notation priority, kanji notation priority, and abbreviation priority. The corrected document acquisition prompt 660 requests a corrected document in which the variation in the word representation is corrected with priority given to the headword (see element 661).

[0108] Following the correction of the variation in the word representation, the corrected document acquisition prompt 660 requests the correction of the variation in the end-of-sentence representation (see element 661). The variation in the end-of-sentence representation includes at least one of the variation in tense and the variation between the desu / masu form and the da / de aru form.

[0109] ≪Document Correction Apparatus: Variations of Synonym Acquisition Prompt and Corrected Document Acquisition Prompt≫ FIG. 37 is a diagram showing the synonym acquisition prompt 650A according to the present embodiment. Compared with the synonym acquisition prompt 650 (see FIG. 34), it emphasizes collecting synonyms in consideration of the context. FIG. 38 is a diagram showing the synonym dictionary 550A including meanings according to the present embodiment. FIG. 39 is a diagram showing the corrected document acquisition prompt 660A according to the present embodiment. Compared with the corrected document acquisition prompt 660 (see FIG. 36), it emphasizes correcting the word variation in consideration of the context.

[0110] As described above, the synonym acquisition prompt 650A requests a synonym group considering the meaning in the context. The synonym group includes the meaning in the context. The corrected document acquisition prompt 660A requests a corrected document obtained by correcting the word notation variation in consideration of the meaning in the context.

[0111] ≪Document Correction Device: Document Correction Process≫ FIG. 40 is a flowchart of the document correction process according to the present embodiment. In step S61, the dictionary generation unit 116 acquires the related document 540 from the document database 131.

[0112] In step S62, the dictionary generation unit 116 generates the synonym acquisition prompt 650 (see FIG. 34). Here, the element 652 is the entire text information included in the document 540. In step S63, the dictionary generation unit 116 instructs the answer acquisition unit 118 to send the synonym acquisition prompt 650 to the language model server 210 and receives the synonym dictionary 550 as the answer.

[0113] In step S64, the document correction unit 117 starts a process of repeating steps S65 to S66 for each document acquired in step S61. In step S65, the document correction unit 117 generates the corrected document acquisition prompt 660 (see FIG. 36). Here, the element 662 is the text information included in the document.

[0114] In step S66, the document correction unit 117 instructs the response acquisition unit 118 to send the corrected document acquisition prompt 660 to the language model server 210. The document correction unit 117 receives the configured corrected document as a response and stores it in the document database 131 in association with the document before correction.

[0115] ≪Features of the document correction device≫ The document processing device 100 as a document correction device performs document correction to correct synonymous notations, tenses, and fluctuations in sentence endings. Regarding synonyms, corrections are made by referring to the synonym dictionary 550 generated based on related documents 540 such as documents of the same business (see element 662). Therefore, it is possible to correct words corresponding to the meanings specific to the business and the meanings according to the context in which the words are included.

[0116] ≪Modification example of the document correction device: corrected document acquisition prompt≫ In the above-described embodiment, fluctuations in synonyms, fluctuations in tenses, and fluctuations between the desu / masu style and the da / de aru style are corrected using one corrected document acquisition prompt 660, 660A. They may be corrected by dividing them into a plurality of prompts. By dividing them into a plurality of prompts, it can be expected that the accuracy of correction will be improved.

[0117] ≪Modification example: language model≫ The above-described document processing device 100 sends a prompt to the language model server 210 that provides a language model service to obtain a response. The document processing device 100 itself may be equipped with a language model and query itself. FIG. 41 is a functional block diagram of a document processing device 100A according to a modification example of the present embodiment. The document processing device 100A includes a language model 132. The response acquisition unit 118A uses the language model 132 to output a response with the prompt as an input. By not using the language model server 210 of an external service, the risk of information leakage can be reduced. Also, by using the language model 132 trained according to the business of the self-organization, the accuracy of the response is improved, and it becomes possible to expect an improvement in the accuracy of document evaluation / review.

[0118] As described above, the response acquisition unit 118A provided in the document processing apparatus 100A as a document analysis apparatus acquires the item value acquisition result 430 for the item value acquisition prompt 610 using the language model 132 instead of the language model server 210.

[0119] The response acquisition unit 118A provided in the document processing apparatus 100A as a document evaluation apparatus acquires the evaluation result 450 for the evaluation acquisition prompt 620 and the report 460 for the report acquisition prompt 630 using the language model 132 instead of the language model server 210.

[0120] The response acquisition unit 118A provided in the document processing apparatus 100 as a document evaluation criterion extraction apparatus acquires the check content for the check content acquisition prompt (see the evaluation criterion acquisition prompt 640B) using the language model 132 instead of the language model server 210.

[0121] The response acquisition unit 118A provided in the document processing apparatus 100 as a document correction apparatus acquires the synonym group (synonym dictionary 550) for the synonym acquisition prompt 650 and the corrected document for the corrected document acquisition prompt 660 using the language model 132 instead of the language model server 210.

[0122] ≪Other Variants≫ Although some embodiments and variants of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and furthermore, various changes such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and their variants are included in the scope and gist of the invention described in this specification and the like, and are also included in the invention described in the claims and its equivalent scope.

[0123] ≪Hardware Configuration≫ The document processing apparatuses 100 and 100A according to the above-described embodiments are realized by a computer 900 configured as shown in FIG. 42, for example. FIG. 42 is a hardware configuration diagram showing an example of a computer 900 that realizes the functions of the document processing apparatuses 100 and 100A according to the above-described embodiments. The computer 900 includes a CPU 901, a ROM 902, a RAM 903, an SSD 904, and an input / output interface 905 (described as input / output I / F (Interface) in FIG. 42). Further, the computer 900 includes a communication interface 906 (described as communication I / F in FIG. 42) and a media interface 907 (described as media I / F in FIG. 42). The computer 900 may include an HDD (Hard Disc Drive) instead of the SSD 904, or may include an HDD in addition to the SSD 904.

[0124] The CPU 901 operates based on a program stored in the ROM 902 or the SSD 904, and performs control by the control unit 110 in FIG. 1. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 is started up, a program related to the hardware of the computer 900, and the like.

[0125] The CPU 901 controls an input device 910 such as a mouse or a keyboard, and an output device 911 such as a display or a printer via the input / output interface 905. The CPU 901 acquires data from the input device 910 via the input / output interface 905, and outputs the generated data to the output device 911.

[0126] The SSD 904 stores a program executed by the CPU 901 and data used by the program. The communication interface 906 receives data from another device (not shown) such as a language model server 210 via a communication network and outputs the data to the CPU 901, and transmits data generated by the CPU 901 to another device via the communication network.

[0127] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductor memory tape medium, or a semiconductor memory or the like.

[0128] For example, when the computer 900 functions as the document processing apparatuses 100 and 100A according to the above-described embodiment, the CPU 901 of the computer 900 realizes the functions of the document processing apparatuses 100 and 100A by executing the program 138 (see FIG. 1) loaded onto the RAM 903. The CPU 901 reads and executes the program from the recording medium 912. In addition, the CPU 901 may read a program from another device via a communication network, or may install and execute the program 138 from the recording medium 912 to the SSD 904. Note that the document processing apparatuses 100 and 100A are not limited to the hardware computer 900, and may be in the form of a virtual machine.

Explanation of Signs

[0129] 100 Document processing apparatus (document analysis apparatus, document evaluation apparatus, document evaluation criterion extraction apparatus, document correction apparatus) 111 Document analysis unit 112 Document format generation unit 113 Document evaluation unit 114 Evaluation report unit 115 Evaluation criterion extraction unit 116 Dictionary generation unit 117 Document correction unit 118, 118A Answer acquisition unit 131 Document database 132 Language model 140 Document format database 150 Document evaluation report rule database 160 Evaluation Report Database 210 Language Model Server 410 Tabular Document 420, 420A Format Information Document 430, 430A Item Value Acquisition Result 450, 450A Evaluation Result 460 Report 510 Document 520, 520A Evaluation Criteria 520B Evaluation Criteria (Check Contents) 540 Document 550, 550A Thesaurus (Synonym Group) 560 Revised Document 610 Item Value Acquisition Prompt 612 Element (Format Information) 613 Element (Document Information) 620, 620A, 620B, 620C Evaluation Acquisition Prompt 622 Element (Evaluation Criteria) 623 Element (Document Information to be Evaluated) 624 Element (Question) 630, 630A, 630B, 630C Report Acquisition Prompt 632~635 Elements (Report Example) 640, 640A Evaluation Criteria Acquisition Prompt 640B Evaluation Criteria Acquisition Prompt (Check Contents Acquisition Prompt) 650, 650A Synonym Acquisition Prompt 660, 660A Revised Document Acquisition Prompt

Claims

1. A document evaluation unit that creates an evaluation acquisition prompt for requesting an evaluation result of the evaluation target document information including the correspondence between item identification information and item values for one or more items, and referring to the evaluation criteria related to the items; An answer acquisition unit that transmits the evaluation acquisition prompt to a language model server and acquires the evaluation result directly or indirectly from the language model server; An evaluation report unit that creates a report acquisition prompt for requesting a report in a predetermined format based on the evaluation result, comprising: The answer acquisition unit transmits the report acquisition prompt to the language model server and acquires the report directly or indirectly from the language model server, The request for the evaluation result included in the evaluation acquisition prompt is a request in the form of a question for obtaining the evaluation result, The report acquisition prompt includes an example of the question included in the request in the form of a question, an example of the evaluation result as an answer to the question, an example of the report in the predetermined format for the example of the question and the example of the answer, the request in the form of a question included in the evaluation acquisition prompt, and the evaluation result acquired by the answer acquisition unit. A document evaluation apparatus.

2. The evaluation target document information includes evaluation items that are items whose item values are determined according to the number of items whose item values are included in a predetermined range of one or more predetermined values, The evaluation criteria include criteria for determining the values of the evaluation items, The evaluation acquisition prompt includes a request for counting the number of items whose item values are included in the predetermined range, a request for obtaining a count evaluation item value that is the value of the evaluation item determined according to the counted result, and a request for reporting whether the count evaluation item value matches the described evaluation item value that is the value of the evaluation item included in the evaluation target document information. The predetermined format of the report is a format for reporting a discrepancy when the count evaluation item value does not match the described evaluation item value. The document evaluation apparatus according to Claim 1.

3. The report in the predetermined format is a format including an amendment for correcting the described evaluation item value to the count evaluation item value when the count evaluation item value and the described evaluation item value do not match. The document evaluation apparatus according to Claim 2.

4. The evaluation target document information includes evaluation items that are items whose item values are determined according to the maximum or minimum item value among a plurality of comparison target item values that are the item values, The evaluation criteria including criteria for determining the values of the evaluation items, the evaluation acquisition prompt is, a request to obtain a comparative evaluation item value, which is the value of the evaluation item determined according to the result of comparing the values of the comparison target items, including a request to report whether the comparative evaluation item value matches the described evaluation item value, which is the value of the evaluation item included in the document information to be evaluated, the predetermined form of the report is, a form for reporting a mismatch when the comparative evaluation item value does not match the described evaluation item value The document evaluation device according to claim 1.

5. The report in the predetermined form is, a form including an amendment for correcting the described evaluation item value to the comparative evaluation item value when the comparative evaluation item value and the described evaluation item value do not match The document evaluation device according to claim 4.

6. The evaluation criteria are, including prohibited item identification information, which is the item identification information for which it is prohibited for the item value to be a predetermined value, the evaluation acquisition prompt is, a request to obtain the item value of the prohibited item identification information included in the document information to be evaluated, and a request to obtain prohibited value entry item identification information indicating the item identification information for which the obtained item value is the predetermined value, the predetermined form of the report is, a form including the prohibited value entry item identification information The document evaluation device according to claim 1.

7. The response acquisition unit is, instead of the language model server, using a language model to, obtain the evaluation result for the evaluation acquisition prompt, and obtain the report for the report acquisition prompt The document evaluation device according to claim 1.

8. The document evaluation device, creates an evaluation acquisition prompt that includes evaluation target document information including the correspondence between item identification information and item values for one or more items, and requests an evaluation result of the evaluation target document information based on the evaluation criteria with reference to the evaluation criteria for the items; sends the evaluation acquisition prompt to a language model server and obtains the evaluation result directly or indirectly from the language model server; creates a report acquisition prompt that requests a report in a predetermined form based on the evaluation result; sends the report acquisition prompt to the language model server and obtains the report directly or indirectly from the language model server, and the request for the evaluation result included in the evaluation acquisition prompt is, a request in the form of a question for obtaining the evaluation result, the report acquisition prompt is, Examples of questions included in the requirements of the question form, examples of the evaluation results as answers to the questions, examples of the reports in the predetermined form for the examples of the questions and the answers, the requirements of the question form included in the evaluation acquisition prompt, and the evaluation results obtained from the language model server Document evaluation method

Images