Program, method, and information processing apparatus

Abstract

To provide a program, a method and an information processor for further reducing the labor of the operation of information collection, and for further facilitating examination or conclusion in the case of performing determination based on a law.SOLUTION: The method includes receiving, by a server, an input of a question, searching at least one of a plurality of information sources related to laws based on the received question, instructing, by the server, a large scale language model to summarize a search result of the searching, acquiring, by the server, a summary of the search result from the large scale language model, identifying, by the server, a link that enables reference to an information source serving as a source of the result summarized by the large scale language model, and presenting, by a terminal, the result summarized by the large scale language model and the link that enables reference to the information source serving as the source.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, and an information processing device. [Background technology]

[0002] Various stakeholders, including ordinary consumers and businesses, carry out legal procedures. Users from various positions, including lawyers and other experts, and employees of business companies, conduct research and gather information through searches to make decisions based on legal perspectives. Based on this information, they make decisions on specific matters, such as what should be specified in terms of use and the appropriateness of customer attraction measures implemented in conjunction with the provision of services.

[0003] Patent Document 1 points out, with regard to document searching, that "document corpora, such as those containing legal documents, patent documents, medical journals, etc., are searched using query expressions... Often, users can formulate multiple search queries when researching a particular topic. However, it can be difficult for users to efficiently determine which search query will produce the most relevant search results and how completely that search query will search a particular topic. Therefore, many users do not trust their own document corpus searches and may believe that the search results generated are unreliable or not sufficiently complete."

[0004] Patent document 1 focuses on presenting search results as described above and addresses the issue that "there is a need for an alternative method to graphically display electronic document searches to improve the electronic document search experience."

[0005] Patent Document 1 describes the following: "A Venn diagram including a first circle representing a first document set and a second circle representing a second document set is generated and displayed on a graphic display device," "the first circle overlaps with the second circle in an overlapping area representing common electronic documents present in the first document set and the second document set," "the sizes of the first circle and the second circle reflect the number of electronic documents in the first document set and the second document set, respectively," "the first circle overlaps with the second circle in an overlapping area representing common electronic documents present in the first document set and the second document set," and "in response to a user input, a separation of the first circle and the second circle is indicated on the graphic display device, and a first visualization chart from the first circle and a second visualization chart from the second circle are generated." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-010580 Summary of the Invention [Problem to be solved by the invention]

[0007] On the other hand, even if users search the Internet to consider specific cases and reach a conclusion in order to make decisions as described above, there is no guarantee that articles tailored to individual circumstances are available, and users must make decisions while taking into account various information, such as provisions set out in laws and regulations and interpretations of specific cases set out in court cases, which results in repeated search operations.Users must then review and organize the search results before making a decision, which places a heavy burden on the work of the user.

[0008] Therefore, there is a need for technology that can further reduce the effort required for information gathering when making legal decisions, and make it easier to consider and reach conclusions. [Means for solving the problem]

[0009] According to one embodiment of the present disclosure, there is provided a program for operating a computer having a computer processor, the program causing the computer processor to execute the steps of: accepting input of a question; searching at least one of a plurality of legal information sources based on the accepted question; obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the searching step; identifying links for the results summarized by the large-scale language model that allow reference to the source information sources; and presenting the results summarized by the large-scale language model and the links that allow reference to the source information sources. [Effects of the Invention]

[0010] According to the present disclosure, the effort required for information gathering when making legal decisions can be further reduced, making it easier to consider and reach a conclusion. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing the configuration of the system 1. [Figure 2] FIG. 2 is a diagram showing the configuration of the server 20. As shown in FIG. [Figure 3] FIG. 3 is a diagram showing the configuration of the terminal 10. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing the data structure of the user database 211. As shown in FIG. [Figure 5] FIG. 5 is a diagram showing the data structure of the chat consultation history database 212. As shown in FIG. [Figure 6] FIG. 6 is a diagram showing the data structure of the LLM usage history database 213. [Figure 7] FIG. 7 is a diagram showing the data structure of the prompt database 214. As shown in FIG. [Figure 8]FIG. 8 is a diagram showing the flow of processing in which, in response to a user's input question, multiple information sources are searched, an answer is summarized using a large-scale language model, and the answer is presented to the user in a manner that allows reference to the information sources. [Figure 9] FIG. 9 is a diagram showing the flow of processing for generating answers to user questions using a large-scale language model and outputting a report including the legal provisions and precedents that serve as the basis for the answers. [Figure 10] FIG. 10 shows an example of a screen that displays the results of a search of multiple information sources in response to a user's input question. [Figure 11] FIG. 11 shows an example of a screen that displays an answer to a user's question generated by a large-scale language model, along with the legal provisions that serve as the basis for the answer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. The names and functions of the components are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0013] <Outline of embodiment> <1.1 Overall system configuration> FIG. 1 is a diagram showing the configuration of the system 1.

[0014] 1 includes a legal consultation service server 20, a user terminal 10, a precedent search service server 91, a law search service server 92, a book browsing service server 93, an information media service server 94, an artificial intelligence (large-scale language model) service server 95 (hereinafter also referred to as the "large-scale language model service server 95"), a business operator server 96, an SNS server 97, and a user terminal 10A. These devices are connected for communication via a network 80.

[0015] In the illustrated example, terminals such as terminal 10 and terminal 10A are shown as terminals used by users of the legal consultation service provided by server 20, and each user operates a terminal.

[0016] In this embodiment, each device (terminal device, server, etc.) can also be considered as an information processing device. That is, a collection of devices can be considered as one "information processing device," and system 1 can be formed as a collection of multiple devices. The way in which multiple functions required to realize system 1 according to this embodiment are allocated to one or multiple pieces of hardware can be determined appropriately in consideration of the processing capacity of each piece of hardware and / or the specifications required for system 1.

[0017] The terminal 10 is a device operated by a user. In this embodiment, the terminal 10 is operated by a user who conducts legal research, judgments, etc. The terminal 10 and the terminal 10A have the same functional configuration. The terminal 10 is realized, for example, as follows. · Handheld devices such as smartphones and tablets Desktop PCs (Personal Computers), laptop PCs Wearable devices worn by users (wristwatches, glasses, etc.) The terminal 10 includes a communication IF (Interface) 12 , an input device 13 , an output device 14 , a memory 15 , a storage 16 , and a processor 19 .

[0018] The communication IF 12 is an interface for inputting and outputting signals so that the terminal 10 can communicate with an external device.

[0019] The input device 13 is a device for receiving input operations from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.).

[0020] The output device 14 is a device (such as a display or speaker) for presenting information to the user.

[0021] The memory 15 is for temporarily storing programs and data to be processed by the programs, and is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0022] The storage 16 is for storing data, and is, for example, a flash memory or a hard disk drive (HDD).

[0023] The processor 19 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0024] The server 20 is a device for providing a legal consultation service to a user. In this embodiment, the server 20 receives legal consultation requests from users of the terminals 10 in free text chat format, causes the artificial intelligence (large-scale language model) service server 95 to generate answers to the received legal consultation requests, and returns the generated answers to the users, thereby providing the legal consultation service.

[0025] In this embodiment, the server 20 provides legal consultation services to the following users. Users who handle legal matters, such as the legal department of a business company, and provide legal advice. Users who do not necessarily perform legal work exclusively, such as business divisions of a business company Users who provide legal advice to clients as a professional, such as law firms The server 20 may accept requests for legal consultation from clients to legal experts by matching users who provide legal consultation as experts, such as law firms, with users who request legal consultation from experts, such as business companies or individuals. For example, the content of the consultation may be posted on a bulletin board that can be viewed by third parties, and the expert's response may also be made public. Alternatively, the content of the consultation may be kept private and not disclosed to third parties, allowing the client to consult with the expert. The server 20 stores such client consultation content and the expert's response.

[0026] The server 20 includes a communication IF 22 , an input / output IF 23 , a memory 25 , a storage 26 , and a processor 29 .

[0027] The communication IF 22 is an interface for inputting and outputting signals so that the server 20 can communicate with external devices.

[0028] The input / output IF 23 functions as an interface with an input device for receiving input operations from the user and an output device for presenting information to the user.

[0029] The memory 25 is for temporarily storing programs and data to be processed by the programs, and is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0030] The storage 26 is for storing data, and is, for example, a flash memory or a hard disk drive (HDD).

[0031] The processor 29 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0032] The precedent search service server 91 has a database of precedents and allows users to search for judgments made by courts and the like.

[0033] The server 92 for the legal search service has a database of legal documents and allows users to search for legal documents.

[0034] The book browsing service server 93 is a service that allows users to browse electronic content such as magazines, books, etc. For example, by paying a fixed fee periodically, users can browse electronic books in fields such as law.

[0035] The information media service server 94 provides information services, such as collecting and making available blog posts, Q&A sites, news articles, IR information, etc. The information media service server 94 stores information on interpretations based on laws and regulations, as well as guidelines prepared by government agencies and other organizations to publicize operational rules.

[0036] In addition to the above, the information media service server 94 may also provide services that collect and provide the following information. Whether the organization is (or may be) a party that falls under the conditions for contract termination, such as being an anti-social force. For example, there may be publicly available information, such as a database or news article, that indicates that the organization falls under the above conditions. The reputation of a business company or other organization, such as when consumers have a negative reaction and the information is spread in the news, on social media, etc. For example, information may be spread on social media with negative keywords and a certain number of impressions. Information about business performance, such as sales, revenue (profit amount, profit margin, etc.), and assessment results regarding business continuity (continued losses, possibility of bankruptcy, bond ratings, etc.). The large-scale language model service server 95 is a server that executes language processing tasks using a language model constructed by a learning process including artificial intelligence (AI). An LLM (Large Language Model) is a server that has previously learned a large amount of large-scale data (text data, etc.), such as web content on the Internet, or a large amount of data stored in a specified database, and can execute various language processing tasks by providing the task.

[0037] The large-scale language model service server 95 accepts prompt inputs such as text, images, and voice, and generates and responds to the prompts. Examples of LLMs include GPT-3 and GPT-4 developed by OpenAI, and BERT developed by Google.

[0038] The business's server 96 stores data generated in the course of business activities. This data has access rights, and access to the data is restricted to users belonging to the business and external users not belonging to the business.

[0039] The SNS server 97 provides services that encourage interaction between users, such as a service that allows users to view each other's posts. For example, there are services that allow users to view posts by searching the Internet even if they do not have a user account on the SNS, and services that allow users to view posts by other users if they have a user account.

[0040] <1.2 Functional configuration of server 20> 2 is a diagram showing the configuration of the server 20. As shown in FIG. 2, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0041] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0042] The storage unit 202 stores various databases such as a user database 211, a chat consultation history database 212, an LLM usage history database 213, a prompt database 214, and a legal terminology database 215.

[0043] The user database 211 is a database that manages information about each user, as will be described in detail later.

[0044] The chat consultation history database 212 is a database showing the history of consultations that users have had using the services provided by the server 20. Details will be described later.

[0045] The LLM usage history database 213 is a database showing the history of responses generated by the artificial intelligence service server 95 in response to user inquiries. Details will be described later.

[0046] The prompt database 214 is a database that manages templates of prompts to be sent to the large-scale language model service server 95. Details will be described later.

[0047] The legal term database 215 is a database of legal terms, including, for example, the names of issues set forth in laws and regulations (such as "claims," ​​"debts," "damages," and "risk allocation"), terms set forth in legal precedents, and the like.

[0048] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. By operating in accordance with the program, the control unit 203 performs functions shown as a reception control module 2041, a transmission control module 2042, a user management module 2043, a chat consultation processing module 2044, an LLM usage module 2045, and a learning processing module 2046.

[0049] The reception control module 2041 controls the process by which the server 20 receives signals from external devices in accordance with a communication protocol.

[0050] The transmission control module 2042 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0051] The user management module 2043 is a module for managing information about each user who uses the system 1. Specifically, the user management module 2043 accepts registration of information about each user and updates the user database 211.

[0052] The chat consultation processing module 2044 is a program module that processes legal consultation requests received from users via chat and responds to them.

[0053] Specifically, the chat consultation processing module 2044 receives input of the content of the consultation from the user, causes the large-scale language model service server 95 to generate a response, and updates the chat consultation history database 212 .

[0054] The LLM usage module 2045 is a program module that generates a prompt based on the content of the inquiry entered by the user in free text, and generates an answer by sending the generated prompt to the large-scale language model service server 95.

[0055] Specifically, the LLM usage module 2045 manages a series of texts entered by the user, and updates the LLM usage history database 213 based on prompts sent to the large-scale language model service server 95 and responses from the large-scale language model service server 95.

[0056] The learning processing module 2046 is a program module that trains a trained model that outputs a response to the content of a consultation based on data stored in the server 20 as the chat consultation history database 212 or the like.

[0057] <1.3 Configuration of Terminal 10> FIG. 3 is a diagram showing the configuration of the terminal 10. As shown in FIG.

[0058] As shown in FIG. 3, the terminal 10 includes multiple antennas (antenna 111, antenna 112), communication units (first communication unit 120, second communication unit 121) corresponding to the respective antennas, an input device 130 (including a touch-sensitive device 131), a display 132, an audio processing unit 140, a microphone 141, a speaker 142, a position information sensor 150, a camera 160, a motion sensor 170, a memory unit 180, and a control unit 190. The terminal 10 also has functions and configurations (e.g., a battery for storing power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.) that are not specifically shown in FIG. 3. As shown in FIG. 3, the blocks included in the terminal 10 are electrically connected by a bus or the like.

[0059] The antenna 111 emits a signal emitted by the terminal 10 as a radio wave. The antenna 111 also receives a radio wave from space and provides the received signal to the first communication unit 120.

[0060] The antenna 112 emits a signal emitted by the terminal 10 as a radio wave. The antenna 112 also receives a radio wave from space and provides the received signal to the second communication unit 121.

[0061] The first communication unit 120 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 111 so that the terminal 10 can communicate with other wireless devices. The second communication unit 121 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 112 so that the terminal 10 can communicate with other wireless devices. The first communication unit 120 and the second communication unit 121 are communication modules including a tuner, a received signal strength indicator (RSSI) calculation circuit, a cyclic redundancy check (CRC) calculation circuit, a high-frequency circuit, and the like. The first communication unit 120 and the second communication unit 121 perform modulation / demodulation, frequency conversion, and the like for wireless signals transmitted and received by the terminal 10, and provide the received signals to the control unit 190.

[0062] Input device 130 has a mechanism for accepting input operations by a user. Specifically, input device 130 is configured as a touch screen and includes touch-sensitive device 131. Touch-sensitive device 131 accepts input operations by a user of terminal 10. Touch-sensitive device 131 detects the user's touch position on the touch panel, for example, by using a capacitive touch panel. Touch-sensitive device 131 outputs a signal indicating the user's touch position detected by the touch panel to control unit 190 as an input operation.

[0063] The display 132 displays data such as images, videos, and text under the control of the control unit 190. The display 132 is realized by, for example, an LCD, an organic EL display, or the like.

[0064] The audio processing unit 140 modulates and demodulates audio signals. The audio processing unit 140 modulates a signal provided from the microphone 141 and provides the modulated signal to the control unit 190. The audio processing unit 140 also provides the audio signal to the speaker 142. The audio processing unit 140 is realized, for example, by a processor for audio processing. The microphone 141 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 140. The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal 10.

[0065] The location information sensor 150 is a sensor that detects the location of the terminal 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. The satellite positioning system receives signals from at least three or four satellites, and detects the current location of the terminal 10 equipped with the GPS module based on the received signals.

[0066] The camera 160 is a device that receives light with a light receiving element and outputs the received light as a captured image. The camera 160 is, for example, a depth camera that can detect the distance from the camera 160 to a subject being photographed.

[0067] The motion sensor 170 includes an acceleration sensor, an angular velocity sensor, etc., and detects the movement of the terminal 10 .

[0068] The storage unit 180 is configured with, for example, a flash memory or the like, and stores data and programs used by the terminal 10. The various types of information stored in the storage unit 180 will be described later.

[0069] The control unit 190 controls the operation of the terminal 10 by reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 is, for example, an application processor. By operating in accordance with the program, the control unit 190 fulfills the functions of an operation reception unit 191, a transmission / reception unit 192, a data processing unit 193, a notification control unit 194, and a storage control unit 195.

[0070] Operation acceptance unit 191 performs processing to accept a user's input operation to an input device such as touch-sensitive device 131. Operation acceptance unit 191 determines the type of operation, such as whether the user's operation is a flick operation, a tap operation, or a drag (swipe) operation, based on information about the coordinates where the user has touched touch-sensitive device 131 with a finger or the like.

[0071] The transmitting / receiving unit 192 performs processing for the terminal 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol.

[0072] The data processing unit 193 performs calculations on data that the terminal 10 has received as input in accordance with a program, and outputs the calculation results to a memory or the like.

[0073] The notification control unit 194 performs processing for displaying a display image on the display 132, processing for outputting sound from the speaker 142, and processing for generating vibrations.

[0074] The storage control unit 195 controls the storage of data in the storage unit 180 .

[0075] A description will be given of various types of information stored in storage unit 180. In one aspect, storage unit 180 stores various types of information such as user information 181.

[0076] The user information 181 is information about a user who uses the services of the server 20 .

[0077] <2 Data Structure> 4 is a diagram showing the data structure of the user database 211. The user database 211 includes an item "user ID," an item "name," an item "email address," an item "company ID," an item "department," an item "job title," an item "start date of affiliation," an item "date of leaving employment," and an item "qualifications held."

[0078] The item "user ID" is information that identifies each user.

[0079] The item "Name" is information indicating the name of the user.

[0080] The item "email address" is information about the email address used to contact the user.

[0081] Specifically, the item "email address" includes information about an email address as information for identifying a user, in order to accept the user's login to the service provided by the server 20.

[0082] The item "business ID" is information that identifies the organization to which the user belongs.

[0083] Specifically, the item "business ID" is information for identifying the organization to which the user belongs, and the following organizations are possible. Business companies ·Law firm The item "Department" is information about the department to which the user belongs.

[0084] Specifically, the item "Department" may include the following information regarding the department to which the user belongs: Departments such as the legal department that are expected to carry out legal work - Business divisions, sales divisions, etc. that are not expected to have dedicated legal affairs staff The item "position" is information about the position of the user.

[0085] Specifically, the item "job title" may include the following information as the user's job title: Position with decision-making authority (management, etc.) A position that does not have decision-making authority but demonstrates expertise and a role of assisting the administrative department Not holding a position The item "Affiliation Start Date" is information about the date on which the user became affiliated with the organization.

[0086] Specifically, the item "Affiliation Start Date" may include the following information: Start date if already a member The date when a person is scheduled to join an organization but has not yet joined (it is possible that the person has decided to join the organization but has not yet joined, and therefore has not been assigned permission to view organizational information) · The employee is assigned to the company but the start date has not been finalized (the start date may still be being adjusted) The item "Date of leaving employment" is information about the date on which the user left the organization.

[0087] Specifically, the item "Date of leaving employment" may include the following information: Date of leaving the company (when you leave the company, you may lose the authority to view organizational information) ·Currently employed The item "Qualifications held" is information indicating qualifications held by the user.

[0088] Specifically, the item "Qualifications held" is information for identifying the qualifications held by the user, and may include the following qualifications. · National qualifications such as lawyer, patent attorney, and certified public accountant that demonstrate expertise A qualification recognized by a business, general incorporated association, or other organization as a qualification that indicates expertise 5 is a diagram showing the data structure of the chat consultation history database 212. The chat consultation history database 212 includes an item "consultation thread ID," an item "consultant user ID," an item "post ID," an item "post content," an item "related posts," an item "posting date and time," an item "search target database," an item "search results," an item "LLM output," an item "answer," and an item "user rating."

[0089] The item "Consultation thread ID" is information for identifying each consultation thread in which a user makes a consultation via chat in the legal consultation service provided by the server 20.

[0090] The item "consultant user ID" is information for identifying the user making the consultation.

[0091] Specifically, the item “consultant user ID” corresponds to the item “user ID” of the user database 211.

[0092] The item "Post ID" is information that identifies each post associated with a thread.

[0093] The item "Posted Content" is information indicating the content posted by the user.

[0094] The item "related post" is information indicating other posts related to the post indicated in the item "post ID" for multiple posts managed in a thread.

[0095] Specifically, the item "related post" is information that identifies the most recent post made by a user among multiple posts managed in a thread. In this way, every time a user posts a consultation in a thread indicated by the item "consultation thread ID," the consultation is associated with the item "post ID" of the most recent post made and managed in the chat consultation history database 212, thereby making it possible to manage the order of a series of posts in a thread.

[0096] The item "Posting date and time" is information indicating the timing when the user posted.

[0097] Specifically, the item "posting date and time" may be the time when a question was received from a user in free text or the like.

[0098] The item "database to be searched" is information that specifies the database to be referenced by search when generating an answer by the server 95 of the large-scale language model service.

[0099] Specifically, the item "database to be searched" may include the following information as the database to be referenced. Information provided by the case law search service server 91 Information provided by legal search service server 92 Information provided by the book browsing service server 93 Information provided by the information media service server 94 Information provided by the artificial intelligence (large-scale language model) service server 95 Information provided by the operator's server 96 Information provided by SNS server 97 The item "search results" is information indicating the search results of a search performed on the referenced database based on the user's posted content.

[0100] Specifically, the item "search results" includes the results of a search performed by the server 20 as follows based on the content posted by the user indicated in the item "post ID." Extract words from the user's posted content and use them as search keys. At this time, based on legal terms stored in the legal terminology database 215, legal terms are not broken down into further words and used as search keys. The semantics of user posts is analyzed and vectorized using techniques such as Word2Vec. Documents in the database to be referenced are also vectorized in the same way, and documents stored in the database are searched for based on the proximity of the vectors. The item "LLM output" is information indicating the results generated by the large-scale language model service server 95 based on user posts.

[0101] Specifically, the item "LLM output" includes the result of having the large-scale language model service server 95 generate an answer as follows. The results of searching each database shown in the "Search Results" section are summarized by the large-scale language model service server 95. When generating an answer, the background information such as "a user from the legal department will make the decision" was included in the prompt, and the answer was generated by the large-scale language model service server 95. The item "answer" is information indicating the answer presented to the user.

[0102] Specifically, for the item "answer", an answer including the result generated by the large-scale language model service server 95 and information generated by the server 20 may be presented to the user as follows. - Responses including links to the sources of information that can be used to refer to the results summarized by the large-scale language model service server 95 The server 20 adds the original text of the provisions stored in the legal database and the precedents stored in the precedent database to the results summarized by the server 95 of the large-scale language model service, and provides the answer. The item "user evaluation" is information indicating the user's evaluation of the answer presented by the server 20.

[0103] 6 is a diagram showing the data structure of the LLM usage history database 213. The LLM usage history database 213 includes an item "usage ID," an item "consultation thread ID," an item "posting ID," an item "prompt," an item "LLM output result," and an item "usage date and time."

[0104] The item "Usage ID" is information that identifies each history of answers generated by the server 95 of the large-scale language model service.

[0105] The item "consultation thread ID" is information for identifying a thread in which a user seeks advice in the legal consultation service provided by the server 20.

[0106] Specifically, the item “consultation thread ID” corresponds to the item “consultation thread ID” in the chat consultation history database 212 .

[0107] The item "Post ID" is information that identifies a post of a question entered by a user, for which an answer is to be generated by the large-scale language model service server 95.

[0108] Specifically, the item “Posting ID” corresponds to the item “Posting ID” in the chat consultation history database 212 .

[0109] The item "prompt" is information indicating the content of the prompt applied when sending to the server 95 of the large-scale language model service.

[0110] Specifically, the item "prompt" may include the content of a prompt that is defined as follows and sent to the server 95 of the large-scale language model service. A prompt template defined in the prompt database 214 (to be described later) is added to the search results (item "search results" in the chat consultation history database 212) obtained by searching each database based on the content of the question entered by the user, and the result is sent to the server 95 of the large-scale language model service. The item "LLM output result" is information indicating the content output by the server 95 of the large-scale language model service by sending a prompt to the server 95 of the large-scale language model service.

[0111] The item "Date and time of use" is information on the date and time when the answer was generated by the server 95 of the large-scale language model service.

[0112] 7 is a diagram showing the data structure of the prompt database 214. The prompt database 214 includes the following items: a "prompt ID," an "operator ID," a "prompt template," an "authority setting," and an "availability."

[0113] The item "prompt ID" is information that identifies each prompt.

[0114] The item "Business ID" is information that identifies the organization that uses Prompt.

[0115] Specifically, the item “business ID” corresponds to the item “business ID” in the user database 211.

[0116] The item "prompt template" is information indicating the contents of the prompt template.

[0117] Specifically, the item "prompt template" may include a prompt template containing the following content: A format in which an answer is generated by the large-scale language model service server 95. For example, there may be a format in which items to be included in the answer are specified, and there may be a prompt instructing the server to generate an answer according to the specified items. Such items may include legal issues, a list of stakeholders, generating answers separately for each stakeholder, advantages and disadvantages of each stakeholder, etc. - Instructions to generate summaries for data that is not disclosed to third parties and is referenced by users belonging to an organization, such as data stored on a business's server 96 The item "Usage authority setting" is information indicating the range of users who have the authority to use the prompt to cause the large-scale language model service server 95 to generate an answer.

[0118] Specifically, the item "Usage authority setting" may include information on the authority set as follows: When generating a summary of data that is not disclosed to third parties, such as data stored in a business's server 96, users who do not belong to that organization do not have the authority, and only users who belong to the organization have the authority. - Set permissions according to the user's department (for example, generate summaries for data that can be accessed by a specific department, such as corporate planning) - Set permissions based on user role (for example, generate summaries for data that can be accessed by users with specific roles, such as management) The item "Availability" is information indicating whether or not the prompt can be used as a template.

[0119] Specifically, the "Availability" item includes information on whether or not the prompt template is permitted to be used. For example, the large-scale language model service server 95 may update a prompt template to generate a better answer and stop using the previous prompt template.

[0120] <3 operations> FIG. 8 is a diagram showing the flow of processing in which, in response to a user's input question, multiple information sources are searched, an answer is summarized using a large-scale language model, and the answer is presented to the user in a manner that allows reference to the information sources.

[0121] In step S821, the server 20 presents the user with an operation screen for accepting input of a question in a chat format.

[0122] In step S811, the terminal 10 displays an operation screen and accepts a question input from the user.

[0123] In step S823, the server 20 searches at least one of a plurality of information sources based on the received question, and updates the chat consultation history database 212 based on the search results.

[0124] Specifically, the server 20 searches the database based on the received query. Case law database (Case Law Search Service Server 91), Database of laws and regulations (Server 92, a legal search service), Legal book database (book browsing service server 93), Public comments on legislation (Information Media Services Server 94), Guidelines on Laws and Regulations (Server 94 for Information Media Services), Expert answers to questions stored in a legal advice service (server 20); Data that can be referenced through internet searches (including information such as posts by each user stored on the SNS server 97) The search will cover at least one of the following information sources.

[0125] In addition to the above, the server 20 may also search information sources that are not disclosed to third parties. For example, the server 20 may search data stored in the organization to which the user belongs (the business operator's server 96), such as organizational regulations, organizational Q&As, content posted on the organization's communication tools, or disciplinary cases within the organization, as information sources. The server 20 may search data stored in the organization to which the user belongs as information sources, depending on the authority defined for the user's organization, department, and position shown in the user database 211. For example, the server 20 may determine whether the user has permission to access data stored in the organization based on account information such as the user's email address, and perform a search without targeting information sources that the user does not have permission to view.

[0126] Regarding the search method, server 20 stores information on legal terms in legal terminology database 215 in storage unit 202. Server 20 breaks down the received question sentence into words. At this time, server 20 breaks down legal terms in legal terminology database 215 into words without further breaking them down. Server 20 performs a search based on the broken down words that include legal terms. In step S825, the server 20 generates a prompt to summarize the search results obtained in step S823, instructs the large-scale language model service server 95, and updates the chat consultation history database 212 and the LLM usage history database 213.

[0127] Specifically, the server 20 may include the search results obtained in step S823 in a default prompt and send it to the large-scale language model service server 95, or may generate a prompt using a prompt template stored in the prompt database 214.

[0128] In step S827, the server 20 acquires a summary of the search results from the large-scale language model service server 95 as information generated by the large-scale language model service server 95, and updates the chat consultation history database 212.

[0129] To generate a response to provide to the user, the server 20 may do the following:

[0130] (i) The amount of information in the response must be above a certain level. For example, if the results summarized by the server 95 of the large-scale language model service do not contain a certain amount of information (for example, the number of characters is below a certain level), the server 20 may repeat the search in step S823 and the acquisition of a summary of the search results in step S827 until the results summarized by the server 95 of the large-scale language model service contain a certain amount of information, thereby acquiring a summary of the search results with a certain amount of information.

[0131] (ii) prescribing the format of the response; For example, the server 20 may prompt the large-scale language model service server 95 to generate a summary in accordance with a specified format in step S825, and the server 20 may then obtain a summary of the search results in accordance with the specified format in step S827.

[0132] Specifically, the server 20 may provide a prompt instructing the user to generate responses by dividing the responses into stakeholder units as a default format, and obtain a summary of the search results that is explained separately for each stakeholder.

[0133] In step S829, the server 20 identifies a link that allows reference to the source of the results summarized by the server 95 of the large-scale language model service.

[0134] Specifically, the server 20 breaks down the results summarized by the large-scale language model obtained in step S827 into words. At this time, the legal terms stored in the legal terminology database 215 may not be further broken down. The server 20 searches the Internet, the case law search service server 91, etc. based on the words obtained by the break down. The server 20 identifies high-priority search results as links. The server 20 updates the chat consultation history database 212 by including the identified links and information generated by the large-scale language model service server 95 in the content of the answer to be presented to the user.

[0135] The server 20 refers to the chat consultation history database 212 and presents the answer to the user.

[0136] In step S813, the terminal 10 displays on the operation screen the results of the summary made by the large-scale language model service server 95 and a link that enables reference to the source information source.

[0137] Specifically, the terminal 10 may present a summary of the search results in a prescribed format. More specifically, the terminal 10 may present a summary of the search results in a prescribed format, with separate explanations for each of the interested parties.

[0138] The terminal 10 may present links to information sources in different ways depending on the source information source.

[0139] FIG. 9 is a diagram showing the flow of processing for generating answers to user questions using a large-scale language model and outputting a report including the legal provisions and precedents that serve as the basis for the answers.

[0140] In step S923, the server 20 defines a prompt to be given to the server 95 of the large-scale language model service based on the received question, and generates an answer to the question by giving the prompt to the large-scale language model.

[0141] (i) Regenerate the answer The server 20 may present an answer to the user as described below, and then define a prompt that instructs the user to search for different grounds depending on the user's operation of the operating member, and generate an answer by providing the prompt to a large-scale language model.

[0142] (ii) Generating answers that take into account the assumed time period The server 20 identifies the legal issue of the received question. For example, the server 20 may identify the issue indicated by the legal terminology by referring to the legal terminology database 215 included in the question. The server 20 may refer to the history of amendments to laws and regulations corresponding to the issue, define a prompt to generate an answer based on an information source that assumes that amendments to laws and regulations have been made regarding the issue, and provide the prompt to the server 95 of the large-scale language model service.

[0143] The server 20 may refer to the history of amendments to laws and regulations corresponding to the legal issues at issue in the received question, define a prompt to generate an answer based on the laws and regulations applicable at the time specified for the question or on information sources such as articles about the applicable laws and regulations, and provide the prompt to the server 95 of the large-scale language model service.

[0144] In step S925, the server 20 identifies at least one of the laws and regulations and legal precedents that serve as the basis for the determination of the answer to be generated.

[0145] Specifically, the server 20 may identify the history of amendments to laws and regulations.

[0146] In addition, the server 20 uses the following as the basis for the generated answer: Terms of use and other contracts created by organizations designated by the user, IR information created by organizations designated by the user, Public articles created by organizations or individuals you specify; Materials written in a language other than the language used by the user; At least one of the above may be specified.

[0147] In step S927, the server 20 obtains data on the provisions of the identified law or data on the description of the case by referring to the database of laws and regulations (server 92 of the law search service) or the database of case precedents (server 91 of the case precedent search service) for at least one of the identified law or case precedents.

[0148] In step S929, the server 20 outputs to the terminal 10 an answer including the answer generated by the large-scale language model service server 95 and at least one of data on the legal provisions or data on the precedents identified in step S927 as the basis for the answer.

[0149] In step S913, the terminal 10 displays on the operation screen the answer generated by the large-scale language model and data on the provisions of the law or data on the judicial precedents identified as the basis for the answer.

[0150] (i) Providing the user with the basis for their answer The terminal 10 may present the answer and the data of the provisions of the law or the data of the judicial precedent separately to the user.

[0151] Specifically, the terminal 10 may also present the revision history of the identified law.

[0152] Specifically, the terminal 10 displays an operation member that accepts an operation to search for grounds different from the presented data of the provisions of the law or the data described in the precedent, and when it accepts a user's operation on the operation member, it may transmit the operation content to the server 20, causing the processing of step S923 to be performed.

[0153] The terminal 10 provides the following as the basis for the response: Terms of use and other contracts created by organizations designated by the user, IR information created by organizations designated by the user, Public articles created by organizations or individuals you specify; Materials created in a language other than the language used by the user or the results of translating such materials, At least one of the above may be presented.

[0154] (ii) Accumulation of training data and training method for trained models The terminal 10 may display an operation member that accepts an operation by the user to evaluate the presented answer. The terminal 10 transmits the operation content to the server 20 in response to the operation on the operation member. The server 20 may store in the storage unit 202 the result of the user's evaluation, the attributes of the user who made the evaluation (such as the "Qualifications" item in the user database 211), and the answer presented to the user by the server 20 in association with each other. The server 20 may train a trained model that generates answers to questions based on the result of the user's evaluation of these answers, the attributes of the user who made the evaluation, and the answer.

[0155] <4 Screen example> FIG. 10 shows an example of a screen that displays the results of a search of multiple information sources in response to a user's input question.

[0156] FIG. 11 shows an example of a screen that displays an answer to a user's question generated by a large-scale language model, along with the legal provisions that serve as the basis for the answer.

[0157] The operation screen 1000 is a screen for accepting legal advice from users in a chat format.

[0158] The account display area 1002 is an area where information about the user's account is displayed.

[0159] In the illustrated example, the account display area 1002 displays the user's name, the organization and department to which the user belongs, the qualifications held, and so on.

[0160] The chat display area 1004 is an area that displays questions entered during legal consultation and answers generated by the server 95 of the large-scale language model service.

[0161] The information source display area 1006 is an area for displaying the information source to be referenced.

[0162] In the illustrated example, the information source display area 1006 displays candidate information sources that can be viewed by third parties (with or without registering to use the service), rather than information sources that can be accessed by users belonging to an organization, such as data stored on the business's server 96, and accepts the user's specification of the information source to be accessed.

[0163] The information source display area 1008 is an area for displaying the information source to be referenced.

[0164] In the illustrated example, the information source display area 1008 accepts designation of an information source to be referenced from among information sources that can be referenced by users belonging to an organization, such as data stored in the business's server 96. Specifically, the following information source designations are accepted: - Files and other data stored in storage Messages stored in messaging tools (email, internal communication systems) for each user The server 20 may be configured to receive settings from the user in advance to link with these systems, messaging tools, etc., and to acquire information from these tools, etc.

[0165] The format specification receiving unit 1010 is an operation member that receives a specification of a format in which the server 95 of the large-scale language model service generates a response.

[0166] Specifically, the format specification receiving unit 1010 may present prompts to the user so that the user can select them, in accordance with the templates of the prompts stored in the prompt database 214 .

[0167] The user question display area 1012 is an area for displaying a question input by the user as the content of the consultation.

[0168] In the illustrated example, the user question display area 1012 accepts input of a question in free text.

[0169] The AI ​​answer display area 1014 is an area that displays an answer generated by the large-scale language model service server 95 in response to a question entered by a user.

[0170] The AI ​​answer display area 1014 corresponds to step S813 in FIG.

[0171] The reference link display area 1016 is an area that displays a link that allows the user to refer to the source information source.

[0172] The reference link display area 1016 corresponds to step S813 in FIG.

[0173] In addition, the example in Fig. 11 displays data on legal provisions or precedents specified as the basis for the answer, which corresponds to step S913 in Fig. 9. The example in Fig. 11 displays that an answer including laws and precedents is being generated for the issue corresponding to the question, with the information after the law was revised as the target.

[0174] The chat input receiving section 1018 is an area where a question can be input by inputting text from the user.

[0175] As shown in the figure, the chat input accepting unit 1018 may display information suggesting what to input in order to prompt the user to input a question. This corresponds to step S811 in FIG.

[0176] Furthermore, as shown in the example of FIG. 10, the chat input receiving unit 1018 suggests that an answer will be generated by referring to the information source specified by the user in the information source display area 1006 or the information source display area 1008 (in the example shown, it indicates that the server 92 of the legal search service, etc., will be referred to).

[0177] The time designation receiving unit 1020 is an operation member that receives a designation of the time related to the user's question.

[0178] In the illustrated example, the time designation receiving unit 1020 suggests to the user that the generated answer may be influenced by applicable laws and regulations depending on the time of the question.

[0179] The shuffle operation receiving section 1022 is an operating member that receives an operation to search for other evidence in addition to the sources displayed in the reference link display area 1016 and provide an answer.

[0180] The shuffle operation receiving unit 1022 may receive, for example, an operation from the user to cause the server 20 to perform a search again for information sources specified by the user in the information source display area 1006 or the information source display area 1008 in FIG. 10 and summarize the search results, or may cause the server 20 to perform a search again for information sources not specified by the user and summarize the search results.

[0181] The evaluation operation receiving unit 1024 is an operation member that receives an operation from the user to evaluate the answers displayed in the AI ​​answer display area 1014 and the reference link display area 1016 .

[0182] <Modification> In addition to the aspects described in the above embodiment, the following may be adopted. (1) Improved reliability of generated answers In the above embodiment, if the answer generated by the large-scale language model service server 95 includes a case law number, a legal provision, a guideline citation, etc. as evidence, the server 20 may check whether the generated answer is an actual case law, law, guideline, etc. by searching databases such as the case law search service server 91, the legal search service server 92, the book browsing service server 93, and the information media service server 94. Specifically, the server 20 determines whether the answer generated by the large-scale language model service server 95 includes a description of the content of the case law and the case law number. For example, case law numbers are assigned according to predetermined rules (e.g., "Heisei XX Year (Gyo-Ke) No. XXXXX"), and extracts descriptions that conform to these rules as the case law number. The server 20 queries the case law search service server 91, etc., based on the case law number to obtain the judgment, abstract, etc. of the case law and determines whether they are similar to or identical to the results generated by the large-scale language model service server 95. For example, the server 20 may determine whether the documents are similar based on the degree of agreement between words contained in the sentences, or may vectorize the meanings of the documents and determine whether the documents are similar based on the distance between the vectors. If the server 20 determines that they are not similar, it may have the large-scale language model service server 95 generate a new answer, or may respond to the user with the number of an actual case that exists as a result of an inquiry to the case law search service server 91, etc. For example, it may search for similar judgments from the case law search service server 91, etc., for the introduction to a case included in the answer generated by the large-scale language model service server 95, and respond with the number of the case that corresponds to the search result. This allows the server 95 of the large-scale language model service to determine whether a generated answer is valid by referring to the original case, etc., even if the answer contains a non-existent case, etc. Furthermore, when a case law number, a legal provision, a guideline source, a reference document, or the like is stated as a basis in an answer generated by the large-scale language model service server 95, the server 20 may perform a check to determine whether the content of the generated answer is stated in the reference document, etc., by issuing a prompt to the large-scale language model service server 95 to cause the check. Based on the check result, the server 20 may evaluate the reliability of the answer generated by the large-scale language model service server 95, and may present the evaluation result to the user. For example, the server 20 may present the generated answer and the basis to the user, and may also present to the user whether or not the basis contains any description as a result of checking the basis. Furthermore, for example, if the server 95 of the large-scale language model service performs the matching and outputs a message indicating that there is no description in the evidence, the server 20 may notify the user that "Answer generation failed" or the like without presenting the user with the answer generated by the server 95 of the large-scale language model service. (2) Specify public information issued by other businesses as the source of information. The server 20 may receive an instruction from a user to generate an answer by the large-scale language model service server 95 by comparing the information with information published by other businesses as a comparison target. The server 20 may send a prompt to the large-scale language model service server 95 including an instruction to summarize the search results of information sources published by the business to be compared designated by the user, such as terms of use, investor relations information such as financial statements for investors, press releases, or blogs published by the business to be compared, and to generate an answer to the question entered by the user by comparing the search results with information sources published by the business to be compared, thereby causing the large-scale language model service server 95 to generate an answer. Furthermore, for example, the server 20 may cause the large-scale language model service server 95 to generate an answer to a question entered by the user, and may also present to the user a summary of the answer based on the information sources of the businesses being compared. This allows users to be provided with information on legal reorganization as well as benchmark cases of others. (3) Specify domestic and international information sources. The server 20 may receive from the user a designation of whether the information is published domestically or overseas (or in a specific country), and may then refer to the information source according to the user's designation (for example, if the user designates Europe, the server 20 may define a prompt to summarize the results of a search for information in Europe) and receive an instruction to have the large-scale language model service server 95 generate an answer. This allows users to be provided with information on legal reorganization as well as overseas cases. (4) Generate answers based on industry information The server 20 may search for content posted by each user on the SNS server 97 and have the large-scale language model service server 95 summarize the results to generate an answer. For example, in a service provided by a social networking service server 97, by referring to the posts of each user's reaction to a particular news article, the large-scale language model service server 95 can generate an answer that takes into account industry conditions, etc. (5) Sharing highly rated answers In the above description of the embodiment, the server 20 receives user evaluations of answers generated by the large-scale language model service server 95, and weights the answers according to the user's attributes (e.g., whether the user is qualified with specialized knowledge, has a certain number of years of work experience, holds a managerial position, etc.). The server 20 may prioritize and display answer results to the user in the form of rankings, etc., according to the evaluation results of these answers. This not only improves the accuracy of the answers, but also makes it easier to present answers with higher expected values ​​and better content that have been evaluated by users in the past to questions the user wants to know. (6) Classify the question to generate an answer using a large-scale language model service, and create prompts based on the classification results. In addition to what has been described in the above embodiment, in order to have the large-scale language model service generate the answer intended by the user, the user's question may be classified, a prompt may be created based on the classified results, and the created prompt may be provided to the server 95 of the large-scale language model service. For example, the server 20 may store a list of categories for classifying questions in advance, and for each category, a standard for causing the large-scale language model service to generate an answer may be prepared. When the server 20 receives a question input from a user, it determines which category in the question list the input question falls into. For example, the server 20 receives the user's question as input and provides a prompt to the server 95 of the large-scale language model service stating, "Please answer which category in the category list the input question falls into." This causes the server 95 of the large-scale language model service to generate an answer indicating which category in the category list the user's question falls into. When the server 20 receives the answer generated by the server 95 of the large-scale language model service, the server 20 references the category list and generates a prompt including "criteria for generating an answer" associated with the corresponding category, and provides the generated prompt to the server 95 of the large-scale language model service. This allows the server 20 to obtain the answer generated by the server 95 of the large-scale language model service for the question received from the user based on the "criteria for generating an answer" indicated in the category list. For example, the server 20 may categorize the user's question and select information resources such as books and legal precedents as "criteria for generating an answer." For example, if it can be determined from the user's question that the user wants to know about the procedure, the answer to the user may not need to include information about legal precedents. Therefore, in this case, the server 20 creates a prompt using the absence of legal precedents as the "criterion for generating an answer." For example, if the user inputs a question such as "Please tell me about points to note regarding a mutually agreed-upon divorce," the server may want to provide an answer about the procedure. Furthermore, if it can be determined from the user's question that the user wants to know about cases or definitions of terms, an answer may be generated based on legal precedents as the "criterion for generating an answer." In this way, by providing a prompt for the classification to which the question applies, rather than sending all the information at once to the large-scale language model service server 95 and having it generate an answer, it may be possible to reduce the amount of computing resources consumed by the large-scale language model service server 95.

[0183] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0184] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory.

[0185] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0186] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0187] <Additional Notes> The matters explained in the above embodiment will be supplemented below.

[0188] <Additional notes on the first embodiment>

[0189] (Appendix 1) A program for operating a computer having a computer processor, the program including the steps of: receiving a question input from a user; searching at least one of a plurality of legal information sources based on the received question; obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the search step; identifying a link for the result summarized by the large-scale language model that allows reference to the source information source; and presenting the result summarized by the large-scale language model and the link for reference to the source information source. A program that executes.

[0190] (Appendix 2) The program described in Appendix 1, wherein in the searching step, the search is performed on at least one of a plurality of legal information sources, and based on the received question, the program searches on at least one of the following information sources: a database of case law, a database of laws and regulations, a database of legal books, public comments on laws and regulations, guidelines on laws and regulations, and expert answers to questions stored in a legal consultation service.

[0191] (Appendix 3) A program as described in any of Appendices 1 to 2, wherein the memory unit stores information on legal terms, and in the search step, the program performs a search based on the received question by breaking down the sentence of the received question into words, without further breaking down the legal terms stored in the memory unit, and performing a search based on the broken down words including the legal terms.

[0192] (Appendix 4) A program described in any of Appendices 1 to 3, wherein in the identifying step, a link that can reference the source information source is identified by decomposing the results summarized by the large-scale language model obtained in the obtaining step, performing a search based on the results obtained by decomposing the summarized results, and identifying high-priority search results as links.

[0193] (Appendix 5) A program described in any of Appendices 1 to 4, in which, in the obtaining step, if the results summarized by the large-scale language model do not have a certain amount of information, the search in the searching step and the summarization of the search results in the obtaining step are repeated until the results summarized by the large-scale language model reach a certain amount of information, thereby obtaining a summary of the search results having a certain amount of information.

[0194] (Appendix 6) A program described in any of Appendices 1 to 5, wherein in the obtaining step, a summary of the search results in accordance with the specified format is obtained by prompting the large-scale language model to generate a summary in accordance with the specified format, and in the presenting step, a summary of the search results in accordance with the specified format is presented.

[0195] (Appendix 7) In the obtaining step, a prompt is given to the large-scale language model to instruct it to generate the language in accordance with a specified format, The program of Appendix 6, wherein in the step of providing prompts to generate answers by dividing the group into stakeholders and obtaining and presenting summaries of search results explained for each stakeholder, the program presents summaries of search results explained for each stakeholder in a specified format.

[0196] (Appendix 8) 8. The program according to any one of appendices 1 to 7, wherein in the presenting step, the link is presented by presenting a link to the information source in different ways depending on the source information source.

[0197] (Appendix 9) A program described in any one of Appendices 1 to 8, in which in the searching step, based on the received question, data accumulated in the organization to which the user belongs, such as internal regulations, internal Q&As, content posted on internal communication tools, or disciplinary cases within the organization, is searched as an information source.

[0198] (Appendix 10) A method executed by a computer having a computer processor, the method comprising the steps of: receiving an input of a question by the computer processor; searching at least one of a plurality of legal information sources based on the received question; obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the searching step; identifying a link for the result summarized by the large-scale language model that can refer to the source information source; and presenting the result summarized by the large-scale language model and the link for referencing the source information source. How to perform.

[0199] (Appendix 11) An information processing device, comprising: a control unit receiving an input of a question; a search for at least one of a plurality of legal information sources based on the received question; a step of obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the search step; a step of identifying a link that allows reference to the source information source for the result summarized by the large-scale language model; and a step of presenting the result summarized by the large-scale language model and the link that allows reference to the source information source. An information processing device that executes the above.

[0200] <Additional notes on the second embodiment>

[0201] (Appendix 1) A program for operating a computer having a computer processor, the program including the steps of: accepting input of a question about a legal issue; defining a prompt to be given to a large-scale language model based on the accepted question, and generating an answer to the question by giving the prompt to the large-scale language model; identifying the basis for the judgment of the answer to be generated, the step of identifying at least one of a law or a legal precedent that serves as the basis; obtaining data on the provisions of the identified law or legal precedent or data on the description of the legal precedent by referring to a database of laws or a database of legal precedents for at least one of the identified law or legal precedent; and presenting the answer generated by the large-scale language model and the data on the provisions of the law or legal precedent that has been identified as the basis for the answer. A program that executes.

[0202] (Appendix 2) A program as described in Appendix 1, in which in the presentation step, the answer and data on the identified legal provisions or data on the precedent are presented to the user separately from the answer and data on the legal provisions or data on the precedent.

[0203] (Appendix 3) A program as described in Appendix 2, in which in the identifying step, at least one of a law or a precedent is identified, thereby identifying the history of amendments to the law, and in the presenting step, data on the provisions of the identified law or data on the descriptions of the precedent is presented, thereby also presenting the history of amendments to the identified law.

[0204] (Appendix 4) A program described in any of Appendices 1 to 3, wherein in the presenting step, an operating member is displayed that accepts an operation to search for grounds different from the presented data of legal provisions or data of precedents, and in the generating step, a prompt that instructs the search for different grounds is defined in accordance with the user's operation of the operating member, and an answer is generated by providing the prompt to a large-scale language model.

[0205] (Appendix 5) A program as described in any of Appendices 1 to 4, wherein in the identifying step, at least one of a contract such as terms of use created by an organization designated by the user, IR information created by an organization designated by the user, a public article created by an organization or individual designated by the user, or a material created in a language other than the language used by the user is identified as the basis for the answer to be generated, and in the presenting step, at least one of a contract such as terms of use created by an organization designated by the user, IR information created by an organization designated by the user, a public article created by an organization or individual designated by the user, a material created in a language other than the language used by the user, or a translation of such a material is presented as the basis for the answer.

[0206] (Appendix 6) A program described in any of Appendices 1 to 5, wherein in the generating step, the program refers to the history of amendments to laws and regulations corresponding to the legal issue in the received question, defines a prompt to generate an answer based on an information source that assumes that amendments to laws and regulations have been made regarding the issue, and provides the prompt to a large-scale language model.

[0207] (Appendix 7) A program as described in any of Appendices 1 to 6, wherein in the receiving step, a specification of a time period for the question is received, and in the generating step, the program refers to a history of amendments to laws and regulations corresponding to the legal issue at issue in the received question, defines a prompt to generate an answer based on the laws and regulations applicable at the time period specified for the question or information sources such as articles about the applicable laws and regulations, and provides the prompt to a large-scale language model.

[0208] (Appendix 8) In the presenting step, an operating member is displayed that accepts an operation by the user to evaluate the presented answer, and the program further causes the computer processor to: a step of causing the evaluation result, the attributes of the user who made the evaluation, and the presented answer to be associated and stored in a memory unit according to the evaluation operation; and a step of training a trained model that generates an answer to a question based on the evaluation result, the attributes of the user who made the evaluation, and the answer that are associated and stored in the memory unit; and a program described in any of Appendices 1 to 7.

[0209] (Appendix 9) A method executed by a computer having a computer processor, the method comprising the steps of: receiving an input of a question about a legal issue by the computer processor; defining a prompt to be given to a large-scale language model based on the received question, and generating an answer to the question by giving the prompt to the large-scale language model; identifying the basis for the judgment of the answer to be generated, wherein at least one of a law or a legal precedent is identified as the basis; obtaining data on the provisions of the identified law or legal precedent or data on the citation of the legal precedent by referring to a database of laws or a database of legal precedents for at least one of the identified law or legal precedent; and presenting the answer generated by the large-scale language model and the data on the provisions of the law or legal precedent identified as the basis for the answer. How to perform.

[0210] (Appendix 10) an information processing device, the information processing device comprising: a control unit of the information processing device receiving input of a question about a legal issue; a step of defining a prompt to be given to a large-scale language model based on the received question and generating an answer to the question by giving the prompt to the large-scale language model; a step of identifying the basis for a judgment of the answer to be generated, the step of identifying at least one of a law or a precedent that serves as the basis; a step of obtaining data on the provisions of the identified law or data on the description of the precedent by referring to a database of laws or a database of precedents for at least one of the identified law or precedent; and a step of presenting the answer generated by the large-scale language model and the data on the provisions of the law or data on the description of the precedent that has been identified as the basis for the answer; An information processing device that executes the above.

Claims

1. A program for operating a computer having a computer processor, the program causing the computer processor to: receiving a question input from a user; a step of searching at least one of a plurality of legal information sources based on the received query; obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the searching step; Identifying links that reference the information sources from which the results summarized by the large-scale language model originate; presenting the results of the summarization by the large-scale language model and a link to the source of the information; A program that executes.

2. In the searching step, the search is performed on at least one of a plurality of legal information sources, Based on the above questions received, Case law database, database of laws and regulations, database of legal books, Public comments on legislation; Guidelines on laws and regulations, Expert answers to questions stored in the legal advice service; 2. The program according to claim 1, wherein the search is performed on at least one of the information sources.

3. The memory unit stores information on legal terms, The program In the searching step, the search is performed based on the received question, breaking down the received question sentence into words, without further breaking down the legal terms stored in the storage unit; The program according to claim 1 , wherein the search is performed based on the decomposed words containing the legal terms.

4. In the step of specifying, a link that allows reference to the information source that is the source is specified, Decomposing the results summarized by the large-scale language model obtained in the obtaining step; conducting a search based on the results obtained by decomposing the summarized results; and identifying a search result with a high priority as the link.

5. 2. The program according to claim 1, wherein, in the obtaining step, if the results summarized by the large-scale language model do not have a certain amount of information, the search in the searching step and the summarization of the search results in the obtaining step are repeated until the results summarized by the large-scale language model reach the certain amount of information, thereby obtaining a summary of the search results having a certain amount of information.

6. In the obtaining step, a summary of the search results is obtained in a prescribed format by prompting the large-scale language model to generate a summary in the prescribed format; 2. The program according to claim 1, wherein the step of presenting comprises presenting a summary of the search results in accordance with the specified format.

7. In the step of obtaining, a prompt is given to the large-scale language model to instruct it to generate a predetermined format, providing the prompt to generate a response by stakeholder to obtain a summary of the search results that is explained for each stakeholder; In the step of presenting, a summary of the search results is presented in accordance with the specified format, The program of claim 6 , wherein the program presents a summary of the search results that is explained separately for each of the stakeholders.

8. In the step of presenting the link, The program according to claim 1 , wherein the program presents a link to the information source in a different manner depending on the information source that serves as the source.

9. In the searching step, Based on the above questions received, The program of claim 1, which searches for data stored in the organization to which the user belongs, such as organizational regulations, Q&As, content posted on communication tools within the organization, or disciplinary cases within the organization, as an information source.

10. 1. A computer-implemented method comprising a computer processor, the method comprising: accepting input of a question; a step of searching at least one of a plurality of legal information sources based on the received query; obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the searching step; Identifying links that reference the information sources from which the results summarized by the large-scale language model originate; presenting the results of the summarization by the large-scale language model and a link to the source of the information; How to perform.

11. An information processing device, comprising: accepting input of a question; a step of searching at least one of a plurality of legal information sources based on the received query; obtaining a summary of the search results by instructing a large-scale language model to summarize the search results of the searching step; Identifying links that reference the information sources from which the results summarized by the large-scale language model originate; presenting the results of the summarization by the large-scale language model and a link to the source of the information; An information processing device that executes the above.

Images