Information processing method, information processing device, and program

Abstract

An information processing method according to one aspect of the present disclosure is executed by a computer 70, and includes: determining one or more phrases that are associated with a keyword relating to an attribute of a document and that may be included in the document (reference sentence determining unit 102a); and, on the basis of an expression in the one or more phrases that is semantically related to the keyword, determining an examination rule for assessing whether or not the expression is included in the document (examination rule determining unit 102b).

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] Conventionally, techniques related to the review of documents such as contracts have been known. For example, Patent Document 1 discloses a technique for reducing the time required for document review confirmation. However, the technique disclosed in Patent Document 1 does not sufficiently improve the efficiency of document review.

[0003] JP 2023-066030 A

[0004] The present disclosure aims to improve the efficiency of document review.

[0005] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, and includes determining one or more phrases that are related to keywords related to attributes of the document and that may be included in the document, and determining an examination rule for determining whether the document includes an expression that is semantically related to the keywords in the one or more phrases.

[0006] Another aspect of the present disclosure is an information processing device that includes a computer, which performs the following steps: determining one or more phrases that are related to keywords related to attributes of the document and that may be included in the document; and determining an examination rule for determining whether the document includes an expression that is semantically related to the keyword in the one or more phrases.

[0007] Another aspect of the present disclosure provides a program that causes a computer to accept input of keywords related to document attributes, and to output screening rules determined based on expressions in one or more sentences that are related to the keywords and that are determined to be included in the document, the screening rules including rules for determining whether the expressions are included in the document.

[0008] Another aspect of the present disclosure is an information processing method executed by a computer, which includes obtaining information regarding at least one of a legal document and keywords related to the provisions of the legal document, inputting instructions to a large-scale language model to generate review rules for reviewing the legal document based on the information, and outputting the review rules obtained from the large-scale language model.

[0009] According to the present disclosure, document review can be made more efficient.

[0010] FIG. 1 is a conceptual diagram of system 1. FIG. 2 is a diagram showing an example of the functional configuration of system 1. FIG. 3 is a diagram showing an example of the operation of reference sentence determination unit 102a. FIG. 4 is a diagram showing an example of the operation of examination rule determination unit 102b. FIG. 5 is a diagram showing an example of the operation of equivalent expression determination unit 102c. FIG. 6 is a diagram showing an example of the operation of regular expression determination unit 102d. FIG. 7 is a diagram showing an example of the operation of evaluation unit 106. A flowchart showing an example of the operation of information processing device 2. A flowchart showing an example of the operation of information processing device 2. A diagram for explaining an example of the hardware configuration of each device of system 1.

[0011] <1. Overview> The review of legal documents, which are an example of documents, is generally performed based on predetermined review rules that correspond to the attributes of the legal documents. Therefore, if there are a wide variety of legal documents to be reviewed, it becomes necessary to establish many review rules accordingly. However, establishing review rules requires a lot of man-hours.

[0012] A system 1 according to this embodiment (hereinafter simply referred to as "system 1") can solve such problems. An example of the operation of system 1 will be described with reference to FIG. 1 . System 1 includes an information processing device 2 and a terminal device 3. Terminal device 3 accepts keywords input from a user and transmits the keywords to information processing device 2 (S1). Based on the received keywords, information processing device 2 determines an examination rule for determining whether a legal document contains a predetermined expression related to the keyword (S2). Information processing device 2 applies the determined examination rule to a target document, which is a legal document to be examined (S3).

[0013] The system 1 of this embodiment can determine the review rules based on keywords. Therefore, the system 1 can reduce the man-hours required to establish the review rules. As a result, document review becomes more efficient.

[0014] In this disclosure, a "legal document" includes any written text and may have legal effect. Examples of legal documents include contracts, invoices, purchase orders, order confirmations, notices, reports, powers of attorney, memoranda, various company regulations, and ordinances. A legal document may be an entire document or a portion of a document (e.g., a paragraph, a sentence, or a provision).

[0015] In addition, in the present disclosure, documents can include various documents, such as legal documents, business plans for businesses, job postings for personnel and recruitment, resumes, application forms, etc. The review rules are configured to contribute to the review of documents to be reviewed.

[0016] The detailed configuration and operation of the system 1 will be described below.

[0017] 2. Functional Configuration The functional configuration of the system 1 will be described with reference to Fig. 2. The system 1 includes an information processing device 2 and a terminal device 3. The information processing device 2 and the terminal device 3 are configured to be able to communicate with each other via a communication network 4.

[0018] [Terminal Device 3] The terminal device 3 in this embodiment is a device used by a user. The terminal device 3 may be, for example, a personal computer, a smartphone, a tablet terminal, etc. The terminal device 3 may include an input device (for example, a mouse, a keyboard, a touch panel, a camera, a microphone, etc.) and an output device (for example, a display, a speaker, etc.). The terminal device 3 can communicate with other devices by executing an application such as a web browser.

[0019] [Information Processing Device 2] The information processing device 2 of this embodiment is a device that acquires keywords and determines examination rules based on the keywords. In one embodiment, the information processing device 2 is a server device when the terminal device 3 is a client device.

[0020] The information processing device 2 of this embodiment includes a control unit 10, a storage unit 12, a network interface unit 14, and a bus 16. The control unit 10, the storage unit 12, and the network interface unit 14 are electrically connected via the bus 16.

[0021] (Control unit 10) The control unit 10 of this embodiment functions as an acquisition unit 100, a determination unit 102, a judgment unit 104, an evaluation unit 106, and an output unit 108 by executing various programs stored in the storage unit 12 described below.

[0022] Acquisition Unit 100 The acquisition unit 100 of this embodiment includes a keyword acquisition unit 100a and a change information acquisition unit 100b.

[0023] Keyword Acquisition Unit 100a The keyword acquisition unit 100a of this embodiment acquires keywords related to the provisions of legal documents. Keywords can also be said to be items that should be or are desirable to be specified in legal documents. Keywords may be any of words, phrases consisting of one or more words, sentences, and texts consisting of one or more sentences. Examples of keywords include "cost burden related to service provision," "scope of compensation for damages," "submission method for deliverables," "possibility of re-contracting," "payment method," "handling in the event of delay," and "contract termination."

[0024] In one embodiment, the keyword is input to the terminal device 3 used by the user. The user may input the keyword by selecting one from a plurality of keywords displayed on the terminal device 3. The user may also input the keyword in a free input format to the terminal device 3.

[0025] Acquiring a keyword by the keyword acquiring unit 100a includes making the keyword processable in the control unit 10. The keyword acquiring unit 100a may acquire the keyword based on a file including the keyword being uploaded to the information processing device 2, or may acquire a keyword input or stored in a device other than the terminal device 3.

[0026] In one embodiment, the keyword acquisition unit 100a determines keywords based on the content of a target document (hereinafter simply referred to as a "target document"), which is a legal document to be reviewed. For example, if the target document is a contract related to a business outsourcing agreement, the keyword acquisition unit 100a may determine at least some of the following as keywords: "contracting parties," "details of the outsourced work," "amount of remuneration," "ownership of intellectual property rights," and "delivery date of the deliverable." In this case, the keyword acquisition unit 100a may determine keywords by applying natural language processing technology to the target document.

[0027] The target document may refer to all documents identified based on a predetermined contract type, or may be a specific single document. For example, the target document may be "all contracts related to outsourcing contracts" or a specific document such as "a contract concluded between Company A and Company B on a certain date." In other words, the screening rule determination unit 102b, which will be described later, may determine screening rules for screening all documents identified based on a predetermined contract type, or may determine screening rules for screening a specific single document.

[0028] In one embodiment, the keyword acquisition unit 100a determines keywords based on the content of another document that has something in common with the target document. The other document that has something in common with the target document may be a contract of the same contract type as the target document, or may be a template of the target document. For example, if the target document is a contract related to a business outsourcing contract, the keyword acquisition unit 100a may acquire keywords based on a template of the contract related to the business outsourcing contract. In this case, the keyword acquisition unit 100a may acquire (extract) keywords by inputting the template of the contract related to the business outsourcing contract into a large-scale language model along with predetermined instructions.

[0029] The change information acquisition unit 100b of this embodiment acquires change information regarding changes to equivalent expressions (described later) that is input by a user to the terminal device 3. The change information may be information regarding a part or the whole of the equivalent expression after the change, or information regarding the difference between the equivalent expression before the change and the equivalent expression after the change.

[0030] In the present disclosure, "obtaining information" includes making the information processable in the control unit 10. "Obtaining information" may mean receiving the information from another device, reading the information from the storage unit 12, or obtaining the information as a result of predetermined processing.

[0031] —Determining Unit 102— The determining unit 102 of this embodiment includes a reference sentence determining unit 102a, an examination rule determining unit 102b, an equivalent expression determining unit 102c, and a regular expression determining unit 102d.

[0032] -Reference sentence determination unit 102a- The reference sentence determination unit 102a of this embodiment determines one or more sentences (hereinafter referred to as "one or more reference sentences") that are related to the keywords acquired by the keyword acquisition unit 100a and that may be included in the legal document.

[0033] An example of the operation of the reference sentence determination unit 102a will be described with reference to FIG. 3. For example, if the keyword is "scope of damages," the reference sentence determination unit 102a determines reference sentences such as "Party A shall be liable only for direct damages incurred by Party B due to a breach of this Agreement," "Party A shall be liable for losses resulting from the loss or corruption of Party B's data," "The two parties shall not be obligated to compensate each other for indirect losses, including lost profits," "Party A shall be liable to indemnify and defend Party B against all damages arising from claims brought against Party B by third parties in connection with the services covered by this Agreement," and "If damages are incurred by Party A due to Party A's intentional or gross negligence, Party A shall be liable to compensate Party B for all indirect damages in addition to direct damages." While FIG. 3 shows an example in which the reference sentences are sentences, the reference sentences may also be, for example, words, phrases, sentences, etc.

[0034] In one embodiment, the reference sentence determination unit 102a determines one or more reference sentences based on a table in which one or more sentences are associated with each of one or more keywords. For example, in a case where a table in which a first sentence and a second sentence are associated with a first keyword and a third sentence and a fourth sentence are associated with a second keyword is stored in the storage unit 12, the reference sentence determination unit 102a may determine the first sentence and the second sentence as one or more reference sentences when the keyword acquisition unit 100a acquires the first keyword, and may determine the third sentence and the fourth sentence as one or more reference sentences when the keyword acquisition unit 100a acquires the second keyword.

[0035] In one embodiment, the reference sentence determination unit 102a determines one or more reference sentences based on a dataset of combinations of keywords and sentences. The dataset can be generated by analyzing multiple documents. At least a portion of the dataset can be generated by, for example, using headings or parts of clauses in a contract, which is a legal document, as "keywords" and using the clauses or parts of the clauses as "sentences" corresponding to the keywords. The dataset can also be referred to as a sample.

[0036] In one embodiment, the reference sentence determination unit 102a determines the reference sentence by inputting an output instruction for a sentence related to a keyword to the natural language processing model. As shown in FIG. 3, if the keyword is, for example, "scope of compensation for damages," the output instruction may be, for example, "Give multiple examples of clauses in a contract that stipulate the 'scope of compensation for damages'." Note that the output instruction may include the entire keyword, a part of the keyword, or not the keyword itself.

[0037] In one embodiment, the natural language processing model includes a large-scale language model. The large-scale language model may be a natural language processing model in which a training model with hundreds of millions of parameters is trained with hundreds of gigabytes or more of training data. Examples of large-scale language models include GPT-3, GPT-4, BERT, and LLaMA.

[0038] The reference sentence determination unit 102a of this embodiment may input an output instruction to a natural language processing model stored in the memory unit 12 of the information processing device 2, or may send an output instruction to a natural language processing model hosted on another server device, for example, via an API.

[0039] ~Examination Rule Determining Unit 102b~ The examination rule determining unit 102b of this embodiment determines examination rules for examining legal documents based on the keywords acquired by the keyword acquiring unit 100a.

[0040] In one embodiment, the examination rule determination unit 102b may determine examination rules for determining whether a legal document contains a phrase with the same or similar meaning as part or all of a keyword. For example, if the keyword is "scope of damages," the examination rule determination unit 102b may determine an examination rule such as "the phrase 'scope of damages,' 'limit of damages,' 'conditions of damages,' 'scope of compensation,' 'limit of compensation,' or 'conditions of compensation' must be included." In this example, "compensation" is a synonym of "damages," which is part of the keyword, and "limit" and "condition" are both synonyms of "scope," which is another part of the keyword. Phrases with the same or similar meaning as keywords may be defined, for example, based on business practices in the industry in which the target document is used, as well as descriptions in the Kojien dictionary and thesaurus.

[0041] In one embodiment, the examination rule determination unit 102b determines an examination rule based on an expression semantically related to a keyword in one or more reference sentences. In one example, the expression semantically related to a keyword may be a phrase having the same or similar meaning as part or all of the keyword. In this case, the examination rule determination unit 102b may determine an examination rule for determining whether a phrase in one or more reference sentences having the same or similar meaning as a keyword is included in the legal document. Even in this case, whether a keyword and a phrase in one or more reference sentences have the same or similar meaning may be defined based on, for example, business practices in the industry in which the target document is used, as well as descriptions in general dictionaries, specialized dictionaries, thesauruses, etc.

[0042] In one embodiment, an expression semantically related to a keyword in one or more reference sentences may be a common expression in the one or more reference sentences. As described above, the reference sentence determination unit 102a determines one or more reference sentences based on a keyword. Therefore, the common expression in the one or more reference sentences may be an expression semantically related to a keyword.

[0043] In one embodiment, the screening rule determination unit 102b determines a screening rule based on a first common expression and a second common expression in one or more reference sentences. In this case, the screening rule may include a rule regarding the target document including both the first common expression and the second common expression, or may include a rule regarding the target document including at least one of the first common expression and the second common expression.

[0044] In one embodiment, the common expressions in one or more reference sentences are identified by a predetermined regular expression.

[0045] Referring to FIG. 4, an example of the operation of the screening rule determination unit 102b when determining a predetermined regular expression will be described. The reference sentences p1-p5 include words with similar meanings, such as "damage," "loss," and "disadvantage." The reference sentences p1-p5 also include words with similar meanings, such as "responsibility" and "obligation." The reference sentences p1-p5 also include words with similar meanings, such as "burden" and "responsibility." Based on this, the screening rule determination unit 102b can determine the screening rule "having a sentence that matches the regular expression re." The regular expression re corresponds to a sentence that includes the words "damage," "loss," or "disadvantage," the words "responsibility" or "obligation," and the words "burden" or "responsibility," in any order.

[0046] In this disclosure, "a common expression in one or more references" is not limited to a specific phrase commonly contained in the one or more references. A "common expression in one or more references" can also be referred to as a trend or pattern common to the one or more references. In other words, if a legal document contains a description of a trend or pattern common to one or more references, the legal document can be said to contain a common expression in the one or more references. If a legal document contains a description of a similar purpose using similar words and phrases as one or more references, the legal document can also be said to contain a common expression in the one or more references.

[0047] Equivalent Expression Determination Unit 102c: The equivalent expression determination unit 102c of this embodiment determines an equivalent expression that can uniquely determine a regular expression. An example of the operation of the equivalent expression determination unit 102c when determining an equivalent expression will be described with reference to FIG. 5. As described above, the regular expression re corresponds to a sentence that includes, in any order, the phrases "damage," "loss," or "disadvantage," the phrase "responsibility" or "obligation," and the phrase "burden" or "responsibility." Therefore, the equivalent expression determination unit 102c determines an equivalent expression ee that corresponds to including, in any order of patterns p1-p6, phrases in group g1 that include "damage," "loss," or "disadvantage," phrases in group g2 that include "responsibility" or "obligation," and phrases in group g3 that include "burden" or "responsibility." Using such equivalent expressions ee, the regular expression re can be uniquely determined. Note that ">>" in Figure 5 indicates the order between groups. For example, "group g1 >> group g2 >> group g3" corresponds to a sentence in which the words contained in group g1, the words contained in group g2, and the words contained in group g3 are written in that order.

[0048] In one embodiment, the equivalent expression is more readable than the regular expression, and may be, for example, a representation of the regular expression divided into groups that define word groups and patterns that define word order (see FIG. 5 ), a representation of the regular expression in a diagram, a representation of the regular expression in a natural language or a language close to it, or a representation of the regular expression in a predetermined program code format.

[0049] Regular Expression Determination Unit 102d In this embodiment, the regular expression determination unit 102d determines a modified regular expression corresponding to the modified equivalent expression based on the modified equivalent expression obtained by applying the modification information acquired by the modification information acquisition unit 100b to the equivalent expression. With reference to FIG. 6 , an example of the operation of the regular expression determination unit 102d to determine a modified regular expression based on the modified equivalent expression will be described. In this example, it is assumed that the user changes the equivalent expression ee to the equivalent expression mee by adding the word "responsibility" to group g2 of the equivalent expression ee shown in FIG. 5 . The regular expression determination unit 102d determines a modified regular expression mre based on the modified equivalent expression mee. The regular expression mre corresponds to a description including, in any order, the words "damage," "loss," or "disadvantage," the words "responsibility," "obligation," or "duty," and the words "burden" or "responsibility."

[0050] - Determination Unit 104 - The determination unit 104 of this embodiment determines whether the target document satisfies the examination rule determined by the examination rule determination unit 102b. For example, the determination unit 104 determines whether the target document satisfies an examination rule that "the target document must contain a word or phrase having the same or similar meaning as part or all of a keyword." The determination unit 104 also determines whether the target document satisfies an examination rule that "the target document must contain a predetermined regular expression." Hereinafter, determining whether the target document satisfies the examination rule is also referred to as "applying the examination rule to the target document."

[0051] -Evaluation Unit 106- In this embodiment, the evaluation unit 106 applies the examination rules determined by the examination rule determination unit 102b to the evaluation legal document, which is the legal document that serves as the basis for evaluation, and evaluates the examination rules. The evaluation legal document can be said to be a legal document that includes items that should be or are desirable to be included in order to specify keywords (e.g., "scope of damages"). The evaluation legal document corresponds to a positive sample. In other words, if an appropriate examination rule has been determined, the evaluation legal document satisfies the examination rule. If the evaluation legal document does not satisfy the examination rule, it can be said that there is room for improvement in the examination rule.

[0052] If the examination rule is "containing a common expression in one or more reference sentences," the evaluation unit 106 evaluates the examination rule based on whether the legal document to be evaluated contains the common expression. In particular, if the common expression in one or more reference sentences is specified by a predetermined regular expression, the evaluation unit 106 evaluates the examination rule based on whether the legal document to be evaluated contains a statement that matches the predetermined regular expression.

[0053] The evaluation unit 106 of this embodiment may output whether the legal document to be evaluated satisfies the review rule as a true / false value, or may output a continuous value (e.g., a score, a percentage, etc.). If the review rule includes multiple rules, the evaluation unit 106 may evaluate the review rule based on whether the legal document to be evaluated satisfies all of the multiple rules, or may evaluate the review rule based on the number of rules that the legal document to be evaluated satisfies.

[0054] An example of the operation of the evaluation unit 106 will be described with reference to FIG. 7 . In this example, the evaluation unit 106 evaluates the examination rule by applying the examination rule "including regular expression mre (see FIG. 6 )" to each of the evaluation legal documents e1-e3. Specifically, the evaluation unit 106 evaluates the examination rule based on whether each of the evaluation legal documents e1-e3 has a description that matches regular expression mre. In this example, the evaluation legal documents e1-e2 have a description that matches regular expression mre. On the other hand, the evaluation legal document e3 does not have a description that matches regular expression mre. Therefore, the evaluation unit 106 may evaluate the evaluation legal document e3 as "the examination rule is inappropriate" because the evaluation legal document e3 did not satisfy the examination rule, or may evaluate the evaluation rule as "the score of the examination rule is 67 points (≒ 2 / 3 × 100 points)" because, of the evaluation legal documents e1-e3, the evaluation legal documents e1-e2 satisfy the examination rule.

[0055] - Output Unit 108 - The output unit 108 of this embodiment includes an examination result output unit 108a, an evaluation result output unit 108b, and an equivalent expression output unit 108c. The examination result output unit 108a outputs the determination result by the determination unit 104. The evaluation result output unit 108b outputs the evaluation result by the evaluation unit 106. The equivalent expression output unit 108c outputs an equivalent expression.

[0056] In the present disclosure, "outputting information" may mean storing the information in a storage device, displaying the information, transmitting image data for displaying the information to another device, outputting the information as sound, transmitting sound data for outputting the information as sound to another device, or transmitting the information to another device. Also, in the present disclosure, transmitting information from a predetermined device to another device is not limited to transmitting the information through direct communication between the predetermined device and the other device, but may mean transmitting the information to the other device via a communication network 4 or transmitting the information to the other device via a device that relays the information.

[0057] (Storage unit 12) The storage unit 12 of this embodiment stores various information that must be stored for the information processing device 2 to operate. The storage unit 12 stores various programs executed by the control unit 10. The storage unit 12 may store information for identifying words and phrases that have the same or similar meaning as the keywords acquired by the acquisition unit 100. The reference sentence determination unit 102a and the examination rule determination unit 102b described above can operate based on this information.

[0058] (Network Interface Unit 14 ) The network interface unit 14 of this embodiment realizes communication with other devices or systems via the communication network 4 .

[0059] [Communication Network 4] The communication network 4 of this embodiment realizes communication between the information processing device 2 and the terminal device 3. The communication network 4 realizes communication based on, for example, the TCP / IP protocol.

[0060] 8 is a flowchart for explaining an example of the operation of the information processing device 2 until the examination rule is determined. First, the information processing device 2 acquires a keyword (S100). Next, the information processing device 2 determines one or more reference sentences based on the keyword (S102, see FIG. 3). Next, the information processing device 2 determines an examination rule based on a common expression in the one or more reference sentences (S104, see FIG. 4).

[0061] [Operations up to the Review of a Target Document Based on Review Rules] Figure 9 is a flowchart for explaining an example of operations up to the application of review rules to a target document by the information processing device 2. First, the information processing device 2 evaluates the review rules determined in S104 by applying the review rules to the legal document to be evaluated (S200). Next, the information processing device 2 displays the evaluation results by transmitting information for displaying the evaluation results to the terminal device 3 (S202). Next, the information processing device 2 displays the equivalent expressions by transmitting information for displaying equivalent expressions of regular expressions included in the review rules to the terminal device 3 (S204, see Figure 5).

[0062] Next, the information processing device 2 acquires the change information input by the user to the terminal device 3 (S206). Next, the information processing device 2 determines a regular expression after the change based on the equivalent expression after the change (S208, see FIG. 6). In this way, the user can change the regular expression by changing the equivalent expression according to the evaluation result of S202.

[0063] Next, the information processing device 2 applies the examination rule to the target document (S210). Next, the information processing device 2 transmits information for displaying the judgment result to the terminal device 3, thereby displaying the judgment result (S212).

[0064] 4. Effects The system 1 of this embodiment can determine review rules based on keywords. This allows the user to efficiently determine review rules. As a result, the review of legal documents becomes more efficient.

[0065] In one embodiment, the one or more reference sentences used to determine the screening rule are output from a natural language processing model (particularly a large-scale language model), thereby reducing the effort required to prepare data, tables, etc. in advance to determine the one or more reference sentences.

[0066] In one embodiment, the screening rules are evaluated by applying them to the evaluation instruments, which allows the quality of the determined screening rules to be quantified.

[0067] In one embodiment, regular expressions included in the screening rules are output as equivalent expressions that can uniquely determine the regular expressions, allowing the user to easily understand what the regular expressions mean.

[0068] In one embodiment, if a user changes the equivalent expression, the change is reflected in the regular expression, allowing the user to easily edit the regular expression.

[0069] The above-described effects are merely examples and do not limit the scope of application of the present disclosure.

[0070] 5. Hardware Configuration An example of a hardware configuration in which the devices included in the above-described system 1 are realized by a computer 70 will be described with reference to Fig. 10. Note that the functions of each device can also be realized by dividing them among multiple devices.

[0071] As shown in FIG. 10, the computer 70 includes a processor 700 , a storage device 702 , an input I / F 704 , a data I / F 706 , a communication I / F 708 , and a display device 710 .

[0072] The processor 700 controls various processes in the computer 70 by executing programs stored in the storage device 702. For example, each functional unit included in the control unit 10 of the information processing device 2 can be realized by the processor 700 executing the programs stored in the storage device 702.

[0073] The storage device 702 is a storage medium such as a RAM (Random Access Memory), etc. The RAM temporarily stores the program code of the program executed by the processor 700 and data required when the program is executed.

[0074] The storage device 702 may also be a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 702 stores an operating system and various programs for implementing the above-described configurations. The storage medium storing the various programs may be a non-transitory computer-readable medium. In addition, the storage device 702 may also store tables that register various information and a database that manages the tables. Such programs and data are loaded into the storage device 702 as needed and referenced by the processor 700.

[0075] The input I / F 704 is a device for receiving input from a user. Specific examples of the input I / F 704 include a camera, a button, a microphone, a keyboard, a mouse, a touch panel, various sensors, a wearable device, etc. The input I / F 704 may be connected to the computer 70 via an interface such as a USB (Universal Serial Bus).

[0076] The data I / F 706 is a device for inputting data from outside the computer 70. A specific example of the data I / F 706 is a drive device for reading data stored in various storage media. The data I / F 706 may be provided outside the computer 70. In this case, the data I / F 706 is connected to the computer 70 via an interface such as a USB.

[0077] The communication I / F 708 is a device for performing data communication with devices external to the computer 70 via the communication network 4, either wired or wirelessly. The communication I / F 708 may be provided external to the computer 70. In this case, the communication I / F 708 is connected to the computer 70 via an interface such as a USB.

[0078] The display device 710 is a device for displaying various types of information. Specific examples of the display device 710 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 710 may be provided outside the computer 70. In this case, the display device 710 is connected to the computer 70 via, for example, a display cable. Furthermore, when a touch panel is used as the input I / F 704, the display device 710 can be configured as an integral part of the input I / F 704.

[0079] Furthermore, the components of the devices included in the system 1 described in the above embodiment are assumed to realize predetermined processing in cooperation with other hardware by the processor 700 executing a program stored in the storage device 702. In other words, these components are envisioned as both software or firmware and the corresponding hardware, and in both of these concepts, they are also referred to as "functions," "means," "parts," "processing circuits," "units," or "modules," and can be interpreted as such.

[0080] 6. Modes of Implementation of the Present Disclosure The above embodiment is merely an example of the present disclosure and does not limit the scope of application of the present disclosure. The system 1 may have a configuration in which at least a portion of the configuration described in the above embodiment is modified and / or expanded, or may not have at least a portion of the configuration described in the above embodiment. For example, the system 1 does not necessarily have to include the equivalent expression determination unit 102c and the equivalent expression output unit 108c described in the above embodiment. The devices included in the system 1 may be physically present devices or so-called virtual machines. Furthermore, the devices included in the system 1 may be devices on the cloud (i.e., computing resources available via a network).

[0081] The devices included in the system 1 may be configured with multiple devices that are physically or logically separated. For example, some of the functions of the information processing device 2 may be implemented in a first information processing device, and other functions may be implemented in a second information processing device that can operate in cooperation with the first information processing device. At least some of the various functions described in the above embodiment, including the keyword acquisition unit 100a, the change information acquisition unit 100b, the reference sentence determination unit 102a, the examination rule determination unit 102b, the equivalent expression determination unit 102c, the regular expression determination unit 102d, the judgment unit 104, the evaluation unit 106, and the output unit 108, may be implemented in the first information processing device, and at least some of the other functions may be implemented in the second information processing device.

[0082] In the above embodiment, the "user" may be an entity that provides the system 1 (e.g., a vendor, etc.), or a user who receives the system 1. The system 1 allows users to easily create screening rules based on criteria not provided by the vendor, criteria that differ in content that is deemed appropriate depending on the business practices of the industry, and criteria related to contract types not anticipated by the vendor. This improves convenience for users.

[0083] In the above embodiment, an example has been described in which keywords are input to a large-scale language model and an examination rule is determined based on one or more reference sentences obtained thereby. However, the present invention is not limited to this. The large-scale language model may be configured to directly output an examination rule for a document (rather than a reference sentence) based on an input of a document or keywords. In this case, instructions such as "Please output an examination rule related to the keyword 'scope of compensation for damages'" or "Please output an examination rule for the 'service outsourcing contract' that is favorable to the outsourcing party" may be input to the large-scale language model.

[0084] Alternatively, the review rules may not be output to the user, but the document may be reviewed based on the review rules, and the review results may be output to the user.

[0085] The large-scale language model described in the above embodiment may be pre-trained. The large-scale language model may be pre-trained to output one or more reference sentences based on keywords, or may be pre-trained to output screening rules based on keywords.

[0086] At least a portion of the various information in the above embodiments may be represented by a vector. For example, when the keyword acquisition unit 100a acquires multiple keywords, the multiple keywords may be expressed as a vector. Similarly, one or more reference sentences determined by the reference sentence determination unit 102a may be expressed as a vector. Similarly, when the screening rule determination unit 102b determines a screening rule including multiple rules, the screening rule may be expressed as a vector of the multiple rules. Furthermore, at least a portion of the various information in the above embodiments may be implemented by replacing it with a feature (or a feature vector) of the information.

[0087] The present disclosure includes, for example, the following embodiments: Note that terms used in the above embodiments are shown in parentheses.

[0088] [Appendix 1] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer 70, and includes determining one or more phrases that are related to keywords relating to attributes of a document (legal document) and that may be included in the document (reference sentence determination unit 102a), and determining an examination rule for determining whether or not an expression is included in the document based on an expression that is semantically related to the keyword in the one or more phrases (examination rule determination unit 102b).

[0089] [Supplementary Note 2] In the information processing method according to Supplementary Note 1, the expression may include a common expression in one or more sentences.

[0090] [Supplementary Note 3] The information processing method described in Supplementary Note 1 may further include determining whether or not an expression is included in a target document that is a document to be reviewed, based on the review rule (the determining unit 104).

[0091] [Supplementary Note 4] In the information processing method according to Supplementary Note 2, at least a part of the one or more phrases may be determined by inputting an instruction to output a sentence related to a keyword to a natural language processing model.

[0092] [Supplementary Note 5] In the information processing method described in Supplementary Note 3, the natural language processing model may include a large-scale language model.

[0093] [Appendix 6] The information processing method described in any one of Appendices 1 to 4 may further include applying an examination rule to an evaluation document (evaluation legal document), which is a document that serves as the basis for evaluation, and evaluating the examination rule based on whether it is determined that the expression is included (evaluation unit 106).

[0094] [Supplementary Note 7] In the information processing method according to any one of Supplementary Note 1 to Supplementary Note 5, the expression may be specified by a predetermined regular expression.

[0095] [Supplementary Note 8] The information processing method according to Supplementary Note 6 may further include outputting an equivalent expression that can uniquely determine the predetermined regular expression (the equivalent expression output unit 108c).

[0096] [Supplementary Note 9] The information processing method described in Supplementary Note 7 may further include acquiring change information regarding a change of an equivalent expression input to a terminal device used by a user (change information acquiring unit 100b), and determining another predetermined regular expression corresponding to the other equivalent expression based on the other equivalent expression obtained by applying the change information to the equivalent expression.

[0097] [Supplementary Note 10] In the information processing method according to any one of Supplementary Note 1 to Supplementary Note 10, the keyword may be input to a terminal device used by a user.

[0098] [Supplementary Note 11] In the information processing method according to any one of Supplementary Note 1 to Supplementary Note 9, obtaining keywords may include determining keywords from a target document that is a document to be examined.

[0099] [Appendix 12] Another aspect of the present disclosure is an information processing method executed by a computer, which includes obtaining information regarding at least one of a legal document and keywords related to the provisions of the legal document, inputting instructions to a large-scale language model to generate review rules for reviewing the legal document based on the information, and outputting the review rules obtained from the large-scale language model.

[0100] [Appendix 13] A method according to another aspect of the present disclosure determines whether a target document, which is a document to be reviewed, contains an expression based on review rules determined by an information processing method described in any one of Appendices 1 to 12.

[0101] [Appendix 14] Another aspect of the information processing device of the present disclosure is an information processing device including a computer 70, which performs the following operations: determining one or more phrases that are related to keywords related to attributes of the document and that may be included in the document (reference sentence determination unit 102a); and determining an examination rule for determining whether or not the document includes an expression that is semantically related to the keyword in the one or more phrases (examination rule determination unit 102b).

[0102] [Appendix 15] Another aspect of the present disclosure provides a program that causes a computer 70 to accept input of keywords related to document attributes, and to output screening rules determined based on expressions in one or more sentences that are related to the keywords and that are determined to be included in the document, the expressions being semantically related to the keywords, the screening rules including rules for determining whether the expressions are included in the document.

[0103] 1...system, 2...information processing device, 3...terminal device, 4...communication network, 10...control unit, 12...storage unit, 14...network interface unit, 70...computer, 100...acquisition unit, 100a...keyword acquisition unit, 100b...change information acquisition unit, 102...determination unit, 102a...reference sentence determination unit, 102b...examination rule determination unit, 102c...equivalent expression determination unit, 102d...regular expression determination unit, 104...judgment unit, 106...evaluation unit, 108...output unit, 108a...examination result output unit, 108b...evaluation result output unit, 108c...equivalent expression output unit

Claims

1. An information processing method executed by a computer, comprising: determining one or more phrases that are related to keywords related to attributes of a document and that may be included in the document; and determining a review rule for determining whether the document contains the expression based on an expression in the one or more phrases that is semantically related to the keyword.

2. The information processing method according to claim 1, wherein the expression includes a common expression in the one or more sentences.

3. The information processing method according to claim 1, further comprising determining whether or not the expression is included in a target document that is a document to be reviewed, based on the review rules.

4. The information processing method according to claim 2, wherein at least a portion of the one or more phrases is determined by inputting an output instruction for a sentence related to the keyword to a natural language processing model.

5. The information processing method according to claim 3, wherein the natural language processing model includes a large-scale language model.

6. The information processing method of claim 1, further comprising applying the review rule to an evaluation document, which is a document that serves as the basis for evaluation, and evaluating the review rule based on whether or not it is determined that the expression is included.

7. The information processing method according to claim 1, wherein the expression is specified by a predetermined regular expression.

8. The information processing method according to claim 6, further comprising outputting an equivalent expression that can uniquely determine the predetermined regular expression.

9. The information processing method according to claim 7, further comprising: acquiring modification information regarding a modification of the equivalent expression input to a terminal device used by a user; and determining another predetermined regular expression corresponding to the other equivalent expression based on the other equivalent expression obtained by applying the modification information to the equivalent expression.

10. The information processing method according to claim 1, wherein the keyword is input to a terminal device used by a user.

11. The information processing method of claim 1, wherein said obtaining includes determining said keywords from a target document, which is a document to be reviewed.

12. An information processing method implemented by a computer, comprising: obtaining information regarding at least one of legal documents and keywords related to provisions of the legal documents; inputting instructions to a large-scale language model to generate screening rules for screening the legal documents based on the information; and outputting the screening rules obtained from the large-scale language model.

13. A method for determining whether or not the expression is contained in a target document, which is a document to be reviewed, based on the review rule determined by the information processing method of claim 1 or 12.

14. An information processing device including a computer, which performs the following steps: determining one or more phrases that are related to keywords related to attributes of a document and that may be included in the document; and determining an examination rule for determining whether or not the document includes an expression that is semantically related to the keyword in the one or more phrases.

15. A program that causes a computer to perform the following steps: accept input of keywords related to document attributes; and output screening rules determined based on expressions in one or more sentences that are related to the keywords and that have been determined to be included in the document, and that are semantically related to the keywords, the screening rules including rules for determining whether the document includes the expressions.

Images