Information processing method and information processing device

Abstract

An information processing method according to an embodiment of the present invention acquires a target document and playbook information including at least a plurality of check criteria including a checkpoint related to the document and a template related to the document, and uses a trained model to generate, on the basis of the playbook information, review information related to the target document based on the checkpoint and / or the template.

Description

Information Processing Method and Information Processing Apparatus

[0001] Embodiments of the present disclosure relate to an information processing method and an information processing apparatus.

[0002] Conventionally, systems for assisting in document review have been proposed. For example, a legal document evaluation method for evaluating legal documents based on evaluation rules has been proposed.

[0003] Japanese Patent Application Laid-Open No. 2019-212315

[0004] The problem to be solved by the present disclosure is to provide a support technique for realizing a more appropriate document review for a document.

[0005] The information processing method according to the embodiment acquires a target document and playbook information including at least a plurality of check criteria including checkpoints related to the document and templates related to the document, and based on the playbook information, uses a learned model to generate review information related to the target document based on at least one of the checkpoints and the template.

[0006] FIG. 1 is a schematic diagram showing an example of a document viewing support system according to the present embodiment. FIG. 2 is a diagram showing an example of a functional block of the information processing apparatus according to the present embodiment. FIG. 3 is a diagram showing an example of a functional block of the user terminal according to the present embodiment. FIG. 4 is a diagram showing an example of a hardware configuration of a computer according to the present embodiment. FIG. 5 is a diagram showing an example of a playbook according to the present embodiment. FIG. 6 is a diagram showing an example of an output of review information according to the present embodiment. FIG. 7 is a diagram showing an example of an output of detailed information of check criteria according to the present embodiment. FIG. 8 is a flowchart showing an example of the flow of review information generation processing.

[0007] Hereinafter, the embodiments will be described in detail with reference to the drawings. In the following description, components having substantially the same functions and configurations are denoted by the same reference numerals, and descriptions will be made only when necessary. Also, the embodiments can be combined with other embodiments, modifications, and prior arts.

[0008] Figure 1 is a schematic diagram showing an example of a document viewing support system 100 according to this embodiment. As shown in Figure 1, it functions as a support system for users when viewing documents. For example, the document viewing support system 100 according to this embodiment comprises at least an information processing device 1, at least one user terminal (terminal device) 3 that can communicate with the information processing device 1 via a network, and an external information processing device 5 that can communicate with the information processing device 1 via a network. The external information processing device 5 may be provided by a vendor other than the vendor that provides the information processing device 1 or the document viewing support system 100 to the user.

[0009] In this embodiment, the user terminal 3 may be one or multiple. Similarly, the user terminal 3 may be one or multiple users. For example, a vendor providing the document viewing support system 100 of this embodiment may issue tenants to multiple organizations such as companies, and one or more users authorized by these organizations may use the document viewing support system 100, including processing documents stored within the tenant issued to the organization and creating and using playbooks.

[0010] Here, the document viewing support system 100 according to this embodiment is a system that supports the viewing (display) of documents on, for example, a browser or a document editing application. In this specification, "document" means any document in any language that is created in accordance with certain rules or standards, such as a rule, law, custom, etc.

[0011] Document data is data related to a document, regardless of its format. While typically text data, it can also include image data, tabular data, and other types of data. Furthermore, a document or document data may include metadata created by word processing software (version number, creation date, update date, author, updater, etc.), formatting information (decorations such as underlining and markers, fonts, indentation, etc.), change history information, comment information, and more.

[0012] In the following explanation, to make the details clearer, we will use the example of a document handled by the document viewing support system 100 being a legal document in Japanese, particularly a contract. However, the legal documents handled by the document viewing support system 100 are not limited to contracts; they may also be, for example, internal organizational regulations. For example, they may be internal regulations in English. Furthermore, the legal documents handled by the document viewing support system 100 may be written in any language.

[0013] Legal documents are, for example, documents that contain legal content that produces a predetermined legal effect, and that include criteria from a predetermined perspective, such as whether the legal content or form is desirable or undesirable. Examples include contracts, application forms, memoranda, internal regulations, policies, etc. In this specification, documents processed by the document viewing support system 100 or legal documents may be referred to as supported documents.

[0014] Furthermore, the document viewing support system 100 according to this embodiment may also include a document creation support function in addition to the document viewing support function. Here, the document creation support function refers to a function for editing (modifying, adding to, deleting, etc.) the content of the document to be supported, and providing information to support editing. In the following, we will use the case in which the document viewing support system 100 includes a document creation support function as an example.

[0015] The services provided by the document viewing support system 100 according to this embodiment (document viewing support services) may include, for example, providing the user with the results of a document review process performed on the document to be supported (review information) on a web browser, performing document editing in response to instructions from the user, saving or providing the edited document to the user, editing additional information associated with the document to be supported (e.g., metadata, comment information, chat information, formatting information, etc.), creating a new document (including a copy), comparing multiple documents (e.g., clearly indicating difference information), automatically proofreading the document to be supported, performing various processes using a translation display function, and adding comments, etc., regarding the case to which the document to be supported belongs. In this specification, editing means, for example, adding, deleting, or changing the information included in the document to be edited.

[0016] Furthermore, in this specification, document review processing refers to processing that analyzes or evaluates the information contained in a supported document based on certain criteria and outputs the results (review information). For example, document review processing includes processing that checks and examines whether the information contained in a supported document is appropriate or not based on certain criteria. It may also include outputting the reasons for determining whether the information contained in the supported document is appropriate or not.

[0017] The document review process specifically includes, for example, comparing the wording (content) in the document with pre-prepared playbook information, and outputting the results of the comparison review or the findings based on the playbook information. Furthermore, the findings based on the playbook information also include outputting the reasons for determining whether the information contained in the supported document is appropriate or not.

[0018] In this specification, information regarding review results generated by LLM7 and information regarding review results provided to and displayed to the user may not be specifically distinguished and will simply be referred to as "review information." Similarly, document data, document files, and documents may not be specifically distinguished and will simply be referred to as "documents," "contracts," etc. Similarly, playbook information and playbooks may not be distinguished and will simply be referred to as "playbook information." The same applies to other elements.

[0019] The document supported by this embodiment may be a document file uploaded from the user terminal 3, a file newly created in the information processing device 1 using an online editor, or a file that has already been uploaded, saved, and edited.

[0020] In this embodiment, the user terminal 3 is a client device managed by a user utilizing the document viewing support service. The user terminal 3, for example, displays documents on the screen of a display device based on information from the information processing device 1. The information processing device 1 and the external information processing device 5 shown in Figure 1 will be described later.

[0021] In this embodiment, the user can use the document review information and editing screen displayed on the display device screen of the user terminal 3 to perform operations such as indentation replacement, document review referencing, comment input, comment referencing, chat information input, chat information referencing, document editing, translation display of the supported document, display of document review information, display of difference information, display of proposed revisions for the supported document, display of points raised, and document saving.

[0022] Specifically, proposed revisions to the supported document include, for example, draft text proposed as revisions to past documents or similar items within documents that are similar to the supported document or a part of the supported document. More specifically, these are clauses proposed as counter-proposals to clauses in contracts that have been reviewed in the past and whose wording is similar to a clause in the supported contract. Another example is a revised draft generated by a Large Language Model (LLM) or similar, which reflects checkpoints, template content, and other check criteria information and document review information in the original draft. A third example is a frequently used draft registered as a "favorite" by the user or the organization to which the user belongs.

[0023] Next, the information processing device 1 shown in Figure 1 will be described using Figure 2. Figure 2 is a diagram showing an example of a functional block of the information processing device 1 according to this embodiment. For example, the functions are realized by the processor of the information processing device 1. The information processing device 1 includes, for example, an acquisition unit 11, a processing unit 12, an output unit 13, and a storage unit 14.

[0024] The functions realized by the acquisition unit 11, processing unit 12, output unit 13, and storage unit 14 are each stored as programs in, for example, main memory or auxiliary storage. The processor can realize the functions related to the acquisition unit 11, processing unit 12, output unit 13, and storage unit 14 by reading and executing the programs stored in main memory or auxiliary storage.

[0025] In this embodiment, the acquisition unit 11 acquires, for example, DOCX format document data as a supported document via a network. The acquisition unit 11 also acquires, for example, document data stored in the storage unit 14 as a supported document. Furthermore, the acquisition unit 11 acquires, for example, document data stored in the storage unit of another device via a network as a supported document.

[0026] The acquisition unit 11 of this embodiment acquires, for example, a document to be supported and playbook information. The playbook information includes at least a plurality of check criteria, and the check criteria of this embodiment include at least checkpoints related to the document and a template related to the document.

[0027] The processing unit 12 of this embodiment performs document review processing using document data of the support target document received from the acquisition unit 11, for example. That is, the processing unit 12 of this embodiment performs review processing on the document data based on playbook information and generates document review information related to the document data. In this specification, the processing unit 12 is also referred to as the review processing unit or the generation unit.

[0028] Furthermore, the document review information generated by the document review process executed by the processing unit 12 of this embodiment is based on information obtained by analyzing the document based on rules or criteria applied to the document, such as playbook information or check criteria (for example, certain rules, laws, or customs).

[0029] For example, review information for a contract like the one in this embodiment may include: (1) whether the clauses included in the document (the contract in this embodiment) are favorable or unfavorable to the user (the party); (2) advice on revising or deleting clauses included in the contract; (3) identification of missing items that should normally be included in the document (contract), and suggestions for items to be added; and (4) information on the importance and degree of recommendation of each item in the review results.

[0030] By using the information described above, users can easily understand the legal risks contained in a contract and review the contract. In addition, the document review information may also include information related to matters other than legal risks, such as formatting information, such as whether the style of the text conforms to standards, and whether the prescribed terminology is used.

[0031] Furthermore, the document review process performed by the processing unit 12 may be performed on support documents that are being created or negotiated, or on support documents that are already completed and whose contents have been finalized. For example, it may include a process to extract risks contained in a concluded contract (an example of a support document that is completed and whose contents have been finalized) (risk check process).

[0032] Here, the risk check process involves using, for example, contract documents between multiple contracting parties (in this embodiment, Company A and Company B), and template data that serves as a standard document for the contract within one of the contracting parties (in this embodiment, Company A), to determine the risks (residual risks) remaining for one of the contracting parties in the supported document, which is a concluded contract, and presenting the results to the user. The risk information data, including the results of the risk check, is an example of document review information.

[0033] The document review process of this embodiment can output, based on playbook information, for example, risks anticipated from clauses, expressions, and wording included in the template but not in the supported document; risks anticipated from clauses, expressions, and wording not included in the template but included in the supported document; risks anticipated from clauses, expressions, and wording included in both the template and the supported document; risks anticipated due to the existence of clauses, expressions, and wording not included in the template or the supported document, etc.

[0034] The processing unit 12 extracts, for example, information contained in the document (document data) that is necessary for generating review information. Various techniques can be used to extract information from the document data. For example, it may be rule-based, such as performing a keyword search on the document data and extracting characters and numbers contained near the keywords. Information can also be extracted by morphological analysis, syntactic analysis, semantic analysis, and contextual analysis. Furthermore, information may be extracted using natural language processing or trained models such as LLM. Note that the extraction of information necessary for generating review information may be performed in advance of the review process and stored in the memory unit. Also, the information extraction may be performed by a system other than this system.

[0035] Furthermore, prior to these processes, information necessary for document review, such as text data, may be extracted from the received document data, or processing such as OCR may be used to extract text data from image data. For example, text data may be extracted from a scanned PDF file of a document. Alternatively, document review processing may be performed on the received document without extracting the information necessary for generating document review information as described above.

[0036] The document review information generation performed by the processing unit 12 can utilize a trained model or employ various other techniques. For example, the document review information may be generated using a rule-based method with the extracted information. Furthermore, a combination of a rule-based method and a trained model can also be employed. In addition, the document may be divided into units such as articles, paragraphs, and clauses before generating the document review information. These divided units may correspond to articles.

[0037] Furthermore, the processing unit 12 generates review information about the target document based on, for example, the playbook information, using at least one of the checkpoints and the template as a reference.

[0038] Furthermore, the processing unit 12 may, for example, generate a prompt based on the playbook information, send the document to be supported and the prompt to the external information processing device 5, and create review information to be displayed on the user terminal 3 from the results received from the external information processing device 5.

[0039] Furthermore, the processing unit 12 may also generate statistical information, for example, regarding the generated review information, that evaluates the effectiveness of the check criteria 20, which represent user evaluations such as good or bad, using multiple review information generated in the past.

[0040] Here, in addition to the evaluation by the user, the statistical information may also include information on whether the document has been revised by referring to multiple evaluation results by the user. Further, the statistical information may include information on the object evaluated by the user. For example, it includes documents, items (such as articles), subsequent versions of the documents or articles (hereinafter also referred to as subsequent articles), differences between the document or article and its subsequent version, and other meta-information, etc.

[0041] Also, when generating review information, in addition to the support target document and playbook information, the processing unit 12 can use statistical information on the effectiveness of the check criteria 20 for the target document in the past to generate review information. Thereby, for example, since review information can be generated by referring to information evaluated as effective by the user, the accuracy of the check criteria and review information displayed for the support target document can be improved.

[0042] Further, the processing unit 12 may further generate, for example, statistical information corresponding to the items of the check criteria 20 in the review information to be generated from the statistical information evaluated in the past.

[0043] The output unit 13 outputs, for example, information for displaying the review information created by the processing unit 12 on the display unit of the user terminal 3 in a display form such as an alert display.

[0044] In the storage unit 14 of the present embodiment, for example, playbooks 200, document data 300, review information 400, and statistical information 500 are stored. The document data 300 includes document data 300 including the support target document and documents. The review information 400 includes, for example, document review information of the support target document and document review information generated in the past.

[0045] Now, let's return to FIG. 1 and explain the external information processing device 5. The external information processing device 5 shown in FIG. 1 includes a learned model such as a large language model (LLM7). The large language model includes, for example, ChatGPT or GPT-4 such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), and further GPT-4o, etc.

[0046] In this embodiment, a prompt is sent from the information processing device 1 to the LLM7 included in the external information processing device 5, the information generated by the LLM7 is received, and based on the information, review information of the support target document is generated in the information processing device 1.

[0047] The learned model may be a single machine learning model or a plurality of machine learning models. For example, through additional learning according to the field of the document to be handled (mainly contract documents and legal fields in this embodiment) or the user to be used, or combination with another learned model that processes input or output, customization or fine-tuning may be performed. Thereby, it becomes possible to generate more appropriate document review information.

[0048] Note that the learned model may be stored in the information processing device 1 provided by the vendor that provides the document viewing support system 100, rather than in the external information processing device 5 provided by an external vendor.

[0049] Next, the details of the user terminal 3 of this embodiment will be described using FIG. 3. FIG. 3 is a diagram showing an example of the functional blocks of the user terminal 3 according to this embodiment. For example, it is a function realized by the processor of the user terminal 3. The user terminal 3 includes, for example, a selection unit 31, a display unit 32, a storage unit 33, and an output unit 34.

[0050] The selection unit 31 of the user terminal 3 in this embodiment enables the user to select or input information. For example, the selection unit 31 allows the user to input and select a target document to be input into the information processing device 1. It also allows the user to select a playbook to be used for reviewing the target document. Specifically, this can be a keyboard, mouse, or voice input device.

[0051] Furthermore, the selection unit 31 can, for example, select an appropriate playbook or check criterion from a list of multiple playbooks or check criteria for reviewing the document to be supported, depending on the type of document.

[0052] The display unit 32 of the user terminal 3 can, for example, display review information 400 generated by the information processing device 1. It can also display document data 300, etc., to the user. Specifically, this could be a display device or the like.

[0053] The storage unit 33 of the user terminal 3 stores programs related to applications that can use the document viewing support system 100. It also stores, for example, document data 300 and playbooks 200. Furthermore, it stores review information 400 acquired from the information processing device 1.

[0054] The output unit 34 of the user terminal 3 can output information selected by the user to the information processing device 1.

[0055] Next, an example of the hardware configuration of a computer 40 that can be used to configure the information processing device 1 and the user terminal 3 will be described using Figure 4. Figure 4 is a diagram showing an example of the hardware configuration of the computer 40 according to this embodiment.

[0056] As shown in Figure 4, the information processing device 1 and user terminal 3 of this embodiment are computers 40, which, as an example, include a processor 41, main memory 42, auxiliary memory 43, input / output interface 44, and communication interface 45. These are interconnected via bus lines 46, which include an address bus, data bus, control bus, etc. Interface circuits (not shown) may be interposed between the bus lines 46 and each hardware resource as appropriate.

[0057] The information processing device 1 and user terminal 3 of this embodiment shown in Figure 4 each have one of each component, but they may also have multiple identical components. Furthermore, although Figures 2 and 3 show one information processing device 1 and one user terminal 3, the software may be installed on multiple computers 40, and each of these computers 40 may execute the same or different parts of the software's processing. In this case, each computer 40 may communicate via a communication interface 45 or the like to execute processing in a distributed computing configuration. In other words, the information processing device 1 and user terminal 3 in this embodiment may be configured as a system that realizes the various functions described later by having one or more computers 40 execute instructions stored in one or more storage devices.

[0058] The various calculations of the information processing device 1 and the user terminal 3 in this embodiment may be performed in parallel using one or more processors 41, or using multiple computers 40 connected via a network. Alternatively, the various calculations may be distributed to multiple processing cores within the processor 41 and executed in parallel. Furthermore, some or all of the processing and means of this disclosure may be performed by at least one of a processor 41 and a storage device located on a cloud that can communicate with the information processing device 1 and the user terminal 3 via a network. Thus, the processing of the document viewing support system 100 in this embodiment may take the form of parallel computing using one or more computers 40.

[0059] The processor 41 may be an electronic circuit (processing circuit, processing circuit, processing circuitry, CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit), etc.) including a control device and an arithmetic unit. Alternatively, the processor 41 may be a semiconductor device including a dedicated processing circuit. The processor 41 is not limited to an electronic circuit using electronic logic elements, but may also be realized by an optical circuit using optical logic elements. Furthermore, the processor 41 may include arithmetic functions based on quantum computing.

[0060] The processor 41 performs calculations based on data and software (programs) input from the internal components of the information processing device 1 and the user terminal 3, and can output calculation results and control signals to the respective devices. The processor 41 may also control the components of the information processing device 1 and the user terminal 3 by executing the OS (Operating System) and applications of the information processing device 1 and the user terminal 3.

[0061] In this embodiment, the information processing device 1 and the user terminal 3 may be implemented by one or more processors 41. Here, the processor 41 may refer to one or more electronic circuits arranged on one chip, or to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0062] In this embodiment, the main memory 42 is a storage device that stores instructions executed by the processor 41 and various data, and the information stored in the main memory 42 is read by the processor 41. The auxiliary storage device 43 is a storage device other than the main memory 42. These storage devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The storage device for storing various data used in the information processing device 1 and the user terminal 3 according to this embodiment may be implemented by the main memory 42 or the auxiliary storage device 43, or by the built-in memory of the processor 41. For example, the storage unit in this embodiment may be implemented by the main memory 42 or the auxiliary storage device 43.

[0063] Multiple processors 41 may be connected to (combined with) one memory device, or a single processor 41 may be connected to it. Multiple memory devices may be connected to (combined with) one processor. In this embodiment, if the information processing device 1 and user terminal 3 consist of at least one memory device and multiple processors 41 connected to (combined with) this at least one memory device, the configuration may include at least one of the multiple processors 41 being connected to (combined with) at least one memory device. This configuration may also be realized by memory devices and processors 41 included in multiple computers. Furthermore, the configuration may include a memory device integrated with a processor 41 (for example, a cache memory including an L1 cache and an L2 cache).

[0064] The input / output interface 44 in this embodiment is an interface such as a USB (Universal Serial Bus) that directly connects to an output device such as a display device, an input device, and an external device. The external device may be a storage device such as a memory device, network storage, or HDD. The external device may also be a device that has some of the functions of the components of the information processing device 1 and the user terminal 3 in this embodiment. In other words, the information processing device 1 and the user terminal 3 may transmit or receive some or all of the processing results of the external device.

[0065] The communication interface 45 in this embodiment is an interface for connecting to a network wirelessly or via a wired connection. The communication interface 45 may be any appropriate interface, such as one that conforms to an existing communication standard. Information may be exchanged with an external device connected via the network through the communication interface 45. The network may be any of the following: WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as information is exchanged between the information processing device 1 and the user terminal 3 or between the information processing device 1 and the external information processing device 5. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication).

[0066] Next, the playbook 200 will be described using Figure 5. Figure 5 shows an example of the playbook 200 according to this embodiment.

[0067] As shown in Figure 5, multiple playbooks 200 (200-1 to 200-M) are stored, for example, in the storage unit 14 of the information processing device 1. A playbook 200 is, for example, information that includes unique criteria set by an organization or individual using the document viewing support service to generate review information 400. The information contained in the playbook in this way may also be referred to as playbook information.

[0068] For example, multiple playbooks 200 are stored to perform review processing according to the type of document. The user can select the playbook 200 corresponding to the target document from among the multiple playbooks 200, depending on the type of document. Alternatively, the selection of the playbook 200 may be performed automatically by the document viewing support system 100. For example, the playbook 200 may be selected based on information attached to the target document (document type, position such as disclosing party or contractor, etc.). The information attached to the target document may be extracted or estimated from the content of the document, or it may be entered by the user.

[0069] One playbook 200 of this embodiment includes a plurality of check criteria 20 (20-1 to 20-N). The check criteria 20 are criteria for checking documents, and include, for example, policies on certain content and information on document templates. Specifically, they include review policies on certain issues in a contract and standard examples of wording to be used for those issues.

[0070] The check criteria 20 of this embodiment include, for example, checkpoints 21 (21-1 to 21-N), recommended policies 22 (22-1 to 22-N), templates (recommended clauses) 23 (23-1 to 23-N), other policies 24 (24-1 to 24-N), other clauses 25 (25-1 to 25-N), etc.

[0071] Checkpoint 21 may be, for example, a custom criterion created by the user. Alternatively, it may be a criterion consisting of pre-defined standard phrases.

[0072] Checkpoint 21 or check criterion 20 may be generated by a machine learning model (trained model) such as LLM based on, for example, recommended clauses, template clauses, or document information. Specifically, it can be generated by sending prompts to LLM such as "Create review checkpoints from the following template" or "Extract the necessary perspectives for reviewing other contract drafts based on the following contract draft." Alternatively, a trained model such as LLM may generate multiple candidate checkpoints 21, from which the appropriate one may be selected and used as the check criterion. An example of checkpoint 21 is information that specifies, for example, "the scope of confidential information is defined."

[0073] Furthermore, the document viewing support system 100 may be equipped with a function that enables the creation of check criteria 20, such as checkpoints 21. Specifically, it receives document files and a list of checkpoints necessary for creating check criteria 20, creates a prompt that enables the creation of check criteria 20 based on the document files, sends the prompt to the LLM, saves the output from the LLM, and saves it in a format that can be used as check criteria 20.

[0074] Recommended policy 22 is information that, for example, defines the background behind the creation of the template and the policies regarding the template. Template 23 is an example of a contract clause that the user has designated as preferable.

[0075] Other policies 24, for example, have a lower priority than recommended policy 22, but are alternative policies that can replace recommended policy 22. For example, they could be policies to propose when contract negotiations are difficult. Other clauses 25, for example, have a lower priority than template 23, but are alternative proposals that can replace template 23. For example, they could be clauses that serve as compromise proposals when contract negotiations are difficult.

[0076] Here is an example of the procedure for creating Playbook 200. First, on the Playbook creation screen, the user enters a name for Playbook 200. The name of Playbook 200 can be freely entered, for example, the contract type such as a consignment contract, the name of the department using it, or the usage scenario. Specifically, for example, a name such as "Nagoya Branch Office_Flyer Ordering_Consignor Side" might be entered. Note that the department name and usage scenario may be assigned as separate items from the Playbook 200 name using custom labels or similar.

[0077] By setting information such as the name and usage scenario of each playbook 200, it becomes easier for the user to find the playbook they should select from among multiple playbook 200 candidates (for example, playbooks of the same category, contract type, and position).

[0078] Next, the category of Playbook 200 is set by selection or input. The category is, for example, the type of document that Playbook 200 will be used to review. For example, if it is a contract, contract types such as service agreement or confidentiality agreement will be selected. The user's role when using Playbook 200 may also be selected. For example, the disclosing party in the contract or the receiving party in the contract will be selected. The selection method may also be, for example, by selecting from pre-defined contract types and user roles using a dropdown menu.

[0079] Next, the check criteria 20 to be included in the playbook 200 are determined. For example, they can be selected from a list of multiple check criteria 20 already stored in the memory unit. As an example, the user selects several check criteria 20 according to the type of document (type of contract) of the playbook to be created. The playbook 200 can be created by adding the selected check criteria 20 to the new playbook 200. The user may select all of the check criteria 20 to be included in the playbook 200, but the document viewing support system 100 may also suggest and recommend them based on the contents of already registered playbooks, settings related to playbooks (such as settings indicating which playbooks should be used preferentially), and information attached to the check criteria 20.

[0080] In the example procedure for creating the playbook 200 described above, the playbook 200 was created using the document viewing support system 100. However, the playbook may also be created and registered by uploading data such as a table summarizing pre-prepared review criteria.

[0081] The check criteria 20 in this embodiment may be created by the user, or it may include standard phrases prepared in advance by the vendor of the document viewing support system 100. Alternatively, it may be newly entered and created by the user. Furthermore, it may be created using other data that compiles pre-prepared review criteria, etc.

[0082] Thus, the playbook 200 may be configured by the user selecting several appropriate check criteria 20 from a plurality of check criteria 20 depending on the type of document. Therefore, it is possible to create a playbook 200 that conforms to unique criteria. Furthermore, it is possible to create an appropriate playbook 200 depending on the type of document.

[0083] Furthermore, regarding the check criteria 20, similar to the creation of the playbook 200, the category and user position of each check criterion 20 may be selected and created or registered at the time of creation. In addition, the check criteria may include an importance level item, as described later. The creation or registration of the check criteria 20 in the document viewing support system 100 can be done, for example, in the same way as the creation or registration of the playbook described above.

[0084] The playbook 200 and check criteria 20 described above can be restricted to users with the necessary privileges, such as the service administrator of this system, for creation and editing. By using a playbook 200 managed and quality-assured by an authorized user, multiple users can conduct appropriate reviews. Furthermore, the service administrator of the playbook 200 can manage permissions, such as granting viewing privileges to users in specific departments.

[0085] If a certain check criterion 20 is adopted in multiple playbooks 200, the check criterion 20 in multiple playbooks 200 may be changed or edited by changing or editing the check criterion 20. This allows for the simultaneous modification of check criterion 20 that broadly relate to the company's policies, such as general clauses that are commonly included in any type of contract, or confidentiality clauses, making it easier for users to maintain the playbooks 200 and review criteria. This can be achieved by assigning an identifiable check criterion ID to each check criterion 20 and managing it so that a certain playbook 200 contains the check criterion 20 with that check criterion ID.

[0086] Next, an example of review information 400 will be described using Figure 6. Figure 6 is a diagram showing an example of the output of review information 400 according to this embodiment.

[0087] Using Figure 6, we will explain the case where the document to be supported is a service contract. The user selects the document to be supported and the corresponding playbook 200 from the user terminal 3, and the information processing device 1 and the external information processing device 5 perform the review process. As a result, the information processing device 1 generates review information 400 as shown in Figure 6. The specific review process will be described later.

[0088] Figure 6 shows an example of review information 400, including document data 300 with highlighted areas to be checked as checkpoints during the review process, and the review results for each of the multiple check criteria 20 included in the playbook 200. For example, the review results for each item of the outsourcing contract, which is document data 300, are shown for each of the check criteria 20.

[0089] For example, in Figure 6, the display area for the first check criterion (hereinafter sometimes simply referred to as "check criterion") 20-1 shows the checkpoint and the review results regarding the presence or absence of risk. Such review results are displayed differently depending, for example, whether the document data 300, which is the document to be supported, meets the requirements of the checkpoint.

[0090] In this embodiment, the first check criterion 20-1 includes a checkpoint statement and a check mark, which is an example of an indicator showing that there is no risk for that checkpoint, to the left of the checkpoint statement.

[0091] Furthermore, the reasons for determining that there is no risk may also be shown. Additionally, selecting an item to display details may reveal templates and other items. For example, in the example screen shown in Figure 6, the reasons for determining no risk and other items are hidden and are displayed when a toggle within the display area of ​​the check criteria is operated. This improves user visibility.

[0092] Next, for example, the second check criterion 20-2 shows the text of the checkpoint and the result of the risk check. In the example of check criterion 20-2, the description in document data 300 differs from the provisions of the checkpoint, meaning that the content of document data 300 does not meet the provisions of the checkpoint, so it is marked with an "×" indicating a risk. Therefore, the reason for determining that there is a risk is written below checkpoint 21.

[0093] In the example screen shown in Figure 6, information regarding check criterion 20-1, which was determined to have no risk, is collapsed and hidden during the toggle, while the reason for the determination regarding check criterion 20-2, which was determined to have a risk, is displayed. By displaying the check criteria differently depending on whether or not there is a risk, it is believed that user visibility will be improved and it will contribute to the proper review of documents.

[0094] Furthermore, check criterion 20-2 also indicates importance. This importance level indicates the importance of check criterion 20. For example, if the risk that may arise with respect to that check criterion is significant and action must be taken if such a risk exists, it will be marked as "High".

[0095] Regarding the second check criterion, 20-2, if you wish to view the details, selecting the "Show Details" option will display the details as shown in Figure 7 below.

[0096] An example of detailed information of the check criterion 20 will be explained using Figure 7. Figure 7 is a diagram showing an example of the display of detailed information of the check criterion 20 according to this embodiment.

[0097] As shown in Figure 7, for example, the details of check criterion 20 will be explained using check criterion 20-2 as an example. The details of check criterion 20 in this embodiment include the name of the checkpoint (in the figure, "Confidentiality Obligation"), the text of the checkpoint 21-2, the importance level of check criterion 20-2 26-2, the review result indicating the presence or absence of risk 27-2 and the reason for it, an example of how to respond if there is a risk 28-2, a recommended policy for the contract clauses 22-2, a recommended clause (template) in line with the recommended policy 22-2 23-2, other policies 24-2, and other clauses 25-2.

[0098] Other policy 24-2 provides an example of a response strategy when an agreement cannot be reached with the other party using the recommended clause (template) 23-2. Additionally, other clause 25-2 provides an example template for when an agreement cannot be reached with the other party using the recommended clause (template) 23-2.

[0099] Furthermore, below the detailed information shown in Figure 7, the name of the playbook used for the review, category 503, position 502, check accuracy indicator 501, and a button image 504 for requesting content revisions are shown.

[0100] Category 503 and Position 502 refer to Category 503 in Playbook 200 and Position 502 in the contract.

[0101] The check accuracy display 501 shown at the bottom of Figure 7 will now be explained. In Figure 6, the check criteria 20 are shown as review information 400 for each item of the document data 300. However, the generated review information 400 may not be sufficiently related to the content of the document data 300.

[0102] Thus, the check accuracy display 501 is provided as a function that allows the user to evaluate whether the content of the document data 300 and the generated review information 400 or check criteria 20 are appropriate, for example, related or not. In this embodiment, the check accuracy display 501 is displayed in an evaluable format for each check criterion 20 shown in Figure 6.

[0103] The check accuracy display 501 is configured to allow the user to select, for example, "Good" if the displayed check criterion 20 corresponds to the content of the document data 300, and "Bad" if the check criterion 20 does not correspond to the content of the document data 300.

[0104] Furthermore, the check accuracy display 501 may be a scoring system, for example, instead of a choice between good or bad. Specifically, it may be a choice between two options, or it may be a selection from multiple levels. Alternatively, it may be information that the user can freely input instead of a selection.

[0105] Such user evaluations of good or bad results against check criterion 20 can be used to assess the effectiveness of check criterion 20. Furthermore, user evaluations of scoring against check criterion 20 can also be used to assess the effectiveness of check criterion 20.

[0106] By collecting user feedback information through such check accuracy indicators 501, the effectiveness of the check criteria 20 can be accumulated as statistical information. Specifically, for example, favorable examples that have been evaluated as good in relation to the check criteria 20, document data 300, and check accuracy indicators 501 can be accumulated. Alternatively, unfavorable examples that have been evaluated as bad can be accumulated. This statistical information, including both favorable and unfavorable examples, can be used as a few-shot example during the review process.

[0107] Furthermore, the statistical information may include not only the effectiveness of the multiple check criteria 20, but also character difference information between document data 300 and subsequent versions of document data 300, and between document data 300 and the articles in subsequent versions of document data 300. The character difference information allows us to infer that the user, based on the judgment result for the relevant article in document data 300, determined that "there is a risk and it needs to be corrected" and made the correction.

[0108] According to this, the clauses in subsequent versions of document data 300 serve as an example of correct data when making revisions based on risk detection results, thus contributing to improved accuracy in generating review information.

[0109] Furthermore, statistical information may also store additional information commented on by users, etc., to the document data 300. The comments may be text written in the document file using the comment function of an application capable of handling document files.

[0110] Furthermore, for example, the review information 400 may display past statistical information regarding the check accuracy display 501 as evaluated by users. That is, the review information 400 may display statistical information regarding the check accuracy display 501 as evaluated by multiple users. For example, statistical information regarding the check accuracy display 501 as evaluated by multiple users could include information such as "how many people have rated it as good, and how many have rated it as bad in the past."

[0111] Furthermore, if a user whose review information has been displayed wishes to modify the contents of Playbook 200 or Check Criteria 20, they can select a button image 504 indicating a modification request. By filling out the modification details in the form that appears and submitting it, they can request modifications from an administrator who has the authority to edit the contents of Check Criteria 20.

[0112] Administrators with the authority to edit playbooks or check criteria can detect the need for revisions based on revision requests from users, and can consider revisions or make edits, including revisions to the content of the playbook or check criteria 20. Furthermore, the number of check accuracy ratings and revision requests are displayed for each check criterion or playbook, allowing administrators to prioritize revision considerations for check criteria or playbooks with poor ratings or a large number of revision requests. When a revision request is received, the document viewing support system 100 can issue a notification to the administrator.

[0113] Next, we will explain the flow of the review information 400 generation process using Figure 8. Figure 8 is a flowchart showing an example of the flow of the review information 400 generation process.

[0114] As shown in Figure 8, the user selects the document data to be supported (explained using a contract as an example) 300 and the playbook 200 corresponding to the contract to be supported 300 from the selection unit 31 of the user terminal 3 (step S1). For example, the user selects the document file containing the contract 300 and / or the playbook 200 stored in the storage unit 33 of the user terminal 3 and sends it to the information processing device 1. Alternatively, the user selects the contract 300 and / or the playbook 200 stored in the storage unit 14 of the information processing device 1.

[0115] The acquisition unit 11 of the information processing device 1 in this embodiment acquires a document file containing document data 300 from the user terminal 3. It also acquires a playbook 200 from the storage unit (step S2).

[0116] The processing unit 12 of the information processing device 1 in this embodiment acquires playbook information from the acquisition unit 11. Based on the playbook information, the processing unit 12 generates a prompt to send to the external information processing device 5. For example, it generates a prompt that instructs the device to generate review information 400 for the contract 300 using the playbook information (step S3).

[0117] One example of a prompt is one that instructs the system to generate review information 400 for each item of the contract 300, based on at least one of the checkpoints 21 of the multiple check criteria 20 included in the playbook and the template 23. For example, a prompt is generated that instructs the system to determine whether each item of the contract 300 corresponds to either the checkpoints 21 of the multiple check criteria 20 included in the playbook information or the template 23.

[0118] Furthermore, the prompt may include content that reflects statistical information or information related to statistical information, such as "Few-Shot-Example".

[0119] Furthermore, the prompt may include supplementary information. Supplementary information is information not included in the playbook information that helps generate appropriate review information 400. For example, if the wording of checkpoint 21 is technical, this information may include explanations of technical terms or simplified wording.

[0120] Furthermore, the prompt may include statistical information containing the results of multiple check accuracy indicators 501. This allows the review information 400 to extract check criteria corresponding to the contract 300 with high accuracy by using the statistical information.

[0121] The output unit 13 of the information processing device 1 transmits a document file containing the prompt generated by the processing unit 12 and the document data 300 of the contract 300 to the external information processing device 5. Alternatively, the command to generate review information and a prompt containing the document data 300 may be transmitted to the external information processing device 5 (step S4).

[0122] The LLM7 of the external information processing device 5 performs a judgment process (review process) for each of the 20 check criteria for the contract 300 based on the prompt and the contract 300 (step S5). Then, it transmits and outputs the review information, including the results of the review process, to the information processing device 1 (step S6).

[0123] The acquisition unit 11 of the information processing device 1 acquires review information from the external information processing device 5. The processing unit 12 of the information processing device 1 creates review information 400 for display to the user based on the determination process (step S7).

[0124] The output unit 13 of the information processing device 1 outputs review information 400 to the user terminal 3 (step S8). The display unit 32 of the user terminal 3 displays the acquired review information 400 (step S9).

[0125] This review information 400 allows users to easily review the contract 300. Furthermore, they can review the contract 300 while referring to the contents of the check criteria 20.

[0126] The information processing method according to this embodiment acquires a target document and playbook information that includes at least a plurality of check criteria, including checkpoints and templates related to the document. Based on the playbook information, it generates review information about the target document using a trained model, with at least one of the checkpoints and templates as the criterion, and outputs the review information to the user terminal.

[0127] This allows the trained model to generate review information based on check criteria, displaying review information to the user, including evaluations and suggested revisions for the target document. Furthermore, since the check criteria included in the playbook information can be freely set by the user, suggested revisions can be generated using playbook information appropriate for the target document. Therefore, users can easily find areas for revision in the target document.

[0128] Furthermore, the target document may be modified to reflect the content of the check criteria. Modifications can be made manually by the user rewriting the target document, or they can be made using a pre-trained model. For example, modifications can be easily made by inputting a prompt into the LLM that can output a proposed modification that reflects the check criteria corresponding to the original document or clause, and obtaining the output. This proposed modification may be automatically replaced in the parts of the document data that should be replaced with the proposed modification. This makes it easy to create appropriate documents.

[0129] Furthermore, in the information processing method according to this embodiment, the review information includes statistical information that evaluates the effectiveness of the check criteria using multiple review information generated in the past.

[0130] This allows for the generation of review information that includes statistical data from past evaluations of the effectiveness of the check criteria, thereby improving the accuracy and effectiveness of the check criteria displayed for the target document.

[0131] Furthermore, in the information processing method according to this embodiment, statistical information that has been evaluated in the past and corresponds to the check criteria is also displayed. This allows the user to utilize the check criteria by referring to the displayed statistical information of past check criteria 20.

[0132] Furthermore, in the information processing method according to this embodiment, the statistical information includes comments attached to the target document. This allows review information to be generated using statistical information that includes check criteria and comments from users, thereby improving the accuracy of the generated review information.

[0133] Furthermore, in the information processing method according to this embodiment, playbook information is created by selecting check criteria from a list of multiple check criteria according to the type of document. Additionally, playbook information can be created independently by the user. This allows for the provision of appropriate review information to the user.

[0134] This allows users to create playbook information based on their own criteria, depending on the type of document, including contracts.

[0135] Furthermore, in the information processing method according to this embodiment, the trained model is a Liberal Machine Learning (LLM). Since LLMs have a higher accuracy than other machine learning models in comparing documents and determining similar parts, or can be easily made to that accuracy, checkpoints in the playbook and the target documents are matched using LLMs, which contributes to the generation of appropriate review information. In addition, since the review information is generated using an LLM capable of generating natural-sounding sentences, it is considered that review information that is easy for users to understand can be obtained.

[0136] The review information displayed to the user as described above is merely an example. Furthermore, information such as checkpoints and templates may be stored separately on multiple storage media, or collected during the review process.

[0137] Although embodiments of this disclosure have been described in detail above, these embodiments are presented as examples only and are not intended to limit the user to individual embodiments. Each embodiment can be modified in various ways, including additions, changes, substitutions, partial deletions, and combinations, without departing from the technical spirit of the present invention. These embodiments and their variations are included within the scope of the invention described in the claims and its equivalents.

[0138] 1... Information Processing Unit 3... User Terminal 5... External Information Processing Unit 7... LLM 11... Acquisition Unit 12... Processing Unit 13, 34... Output Unit 14, 33... Storage Unit 20... Check Criteria 21... Checkpoints 22... Recommended Policy 23... Template (Recommended Clause) 31... Selection Unit 32... Display Unit 40... Computer 41... Processor 42... Main Memory 43... Auxiliary Memory 44... Input / Output Interface 45... Communication Interface 100... Document Viewing Support System 200... Playbook 300... Document Data 400... Review Information 500... Statistical Information 501... Check Accuracy Display

Claims

1. An information processing method that obtains a target document and playbook information that includes at least multiple check criteria, including checkpoints and templates related to the document, and generates review information about the target document based on the playbook information, using a trained model, with at least one of the checkpoints and templates as the criterion.

2. The information processing method according to claim 1, wherein the review information includes statistical information evaluated using a plurality of previously generated review information regarding the effectiveness of the check criteria.

3. The information processing method according to claim 2, wherein the review information further includes the statistical information previously evaluated that corresponds to the check criteria.

4. The information processing method according to claim 2 or 3, wherein the statistical information includes comments attached to the target document.

5. The information processing method according to claim 1, wherein the playbook information is created by selecting a check criterion from a list of multiple check criteria according to the type of document.

6. The information processing method according to claim 1, wherein the trained model is an LLM (Large Language Model).

7. An information processing device comprising: an acquisition unit that acquires a target document and playbook information that includes at least a plurality of check criteria, including checkpoints and templates related to the document; and a processing unit that generates review information about the target document based on the playbook information, using a trained model, with at least one of the checkpoints and templates as the criterion.

Images