Information processing device, method, and program

Abstract

An information processing device is provided that can reduce the cost of checking response messages generated using a large-scale language model. [Solution] Each process is performed using a large-scale language model, and includes a first process of generating a first response statement in response to a response request, a second process of identifying a response strategy for each text portion of the first response statement, a third process of searching a database for each of the strategies to find documents for realizing that strategy, a fourth process of extracting related information for realizing the corresponding strategy from each of the documents, a fifth process of generating a second response statement in response to the response request from the related information corresponding to each of the strategies, and a sixth process of obtaining reference source information for each text portion of the second response statement by referring to the logs of the third and fourth processes.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, method, and program for performing processing using a large-scale language model (LLM). [Background technology]

[0002] Companies are facing the challenge of increasing labor costs for operators and employees who handle customer support and internal inquiries. QA (question and answer)-based chatbots have been introduced, but many of the existing chatbots only respond based on pre-prepared QA scenarios, and unexpected questions require individual responses by humans, which limits the reduction in labor costs.

[0003] Meanwhile, large-scale language models (hereafter referred to as LLMs), which have emerged and developed in recent years as generative AI for text, are expected to be used to generate responses similar to those given by operators to such unexpected questions, and to provide systems that support operators in creating responses. LLMs are not limited to customer support and internal inquiries within companies, but can also be used to automatically generate responses of any content, such as product recommendations. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] GAO, Yunfan, et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023, 2. Summary of the Invention [Problem to be solved by the invention]

[0005] An existing approach for generating responses and other correspondence using LLM is a method called Search Augmentation Generation (hereinafter referred to as RAG), which is disclosed in Non-Patent Document 1, etc. LLM is trained using a huge amount of data and is highly versatile, but applying LLM as is to a specific field (such as customer support or internal inquiries) where the knowledge is not sufficiently contained in the huge amount of data from which it is trained does not necessarily produce appropriate results, and false responses and the like may be obtained as a result of so-called hallucination, which is a known problem with LLM. However, the RAG method makes it possible to simply specialize LLM to a specific field.

[0006] However, even in LLMs that use the RAG method and specialize in specific fields, the occurrence of hallucination is still unavoidable in principle. For this reason, in the operation of LLMs, text output is obtained quickly using generation AI, and this text output is also checked manually as appropriate. It is desirable to improve the efficiency of this manual checking.

[0007] As a concrete means of efficiently manually checking the text output of generative AI, functions that automatically indicate the reference sources that are the information sources for the text output are increasingly being adopted, but the cost of checking is high when simply listing or providing links is simply high.

[0008] 1 is a diagram showing a schematic diagram of how reference sources are indicated according to conventional technology, showing a first conventional example EX101 and a second conventional example EX201. In the first conventional example EX101, after a text body T102 output by a generation AI, multiple links L103 that are reference sources of the text body T102 are listed, but because there is no indication of which parts of the text body T102 are referenced by the multiple links L103, the cost of manual confirmation is high.

[0009] The second conventional example EX201 is an example of "ChatGPT," one of the widely known current LLM services. While this example improves upon the first conventional example EX101 by displaying linking information, the manual verification cost remains high. Specifically, in the second conventional example EX201, three links L203, L204, and L205 are displayed in the text body T202 output by the generation AI, each indicating the source of the text portion. Therefore, the link indicating the source of the text portion in the text body T202 is provided. However, for example, if one link L203 specifically indicates the URL "https: / / aaa..." indicating the location of a website on the Internet as the source of the text portion, actually visiting the website linked to link L203 may result in, for example, a very long piece of text content T206 being displayed. In this case, there is still no indication of which specific partial text T207 in the entire content T206 is the reference source. Therefore, in order to find the partial text T207 that corresponds to the reference source, the user must read the entire text T202 or use trial and error such as keyword search, which increases the verification cost.

[0010] Furthermore, in both the first conventional example EX101 and the second conventional example EX201, it is necessary to manually check the reference source, whose reference method is unknown, and then to check whether or not hallucination has occurred, which further increases the cost of checking.

[0011] In view of the above-mentioned problems of the conventional technology, an object of the present invention is to provide an information processing device, method, and program that can reduce verification costs. [Means for solving the problem]

[0012] To achieve the above object, the present invention is an information processing device that executes the following steps: a first process of generating a first response sentence in response to a response request using a large-scale language model; a second process of identifying a response strategy for each text portion of the first response sentence using the large-scale language model; a third process of searching a database for each of the strategies using the large-scale language model to find documents for implementing the strategies; a fourth process of extracting, from each of the documents, information related to implementing the corresponding strategy; a fifth process of generating a second response sentence in response to the response request from the related information corresponding to each of the strategies using the large-scale language model; and a sixth process of obtaining reference source information for each text portion of the second response sentence by referring to logs of the third and fourth processes using the large-scale language model. The present invention is also characterized by a method and a program corresponding to the device. [Effects of the Invention]

[0013] According to the present invention, by obtaining information on the reference source, the effort required for confirmation can be reduced. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram schematically illustrating how a reference source is specified according to the prior art. [Figure 2] 1 is a configuration diagram of an information processing system according to an embodiment. [Figure 3] FIG. 1 is a configuration diagram of an information processing device according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating the processing of the LLM unit. [Figure 5] 10 is a flowchart illustrating an operation of an information processing device according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating a schematic example of processing by a provisional response generation unit. [Figure 7] FIG. 10 is a diagram illustrating a schematic example of processing by a strategy identification unit. [Figure 8] FIG. 10 is a diagram illustrating a schematic example of processing by a document search unit. [Figure 9]FIG. 10 is a diagram illustrating a schematic example of processing by an information extraction unit. [Figure 10] 10 is a diagram illustrating a schematic example of the processing of a response generation unit and a reference source linking unit. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of a prompt in a reference source linking unit. [Figure 12] FIG. 10 is a diagram illustrating a schematic example of processing by a reference source commentary unit. [Figure 13] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] 2 is a configuration diagram of an information processing system according to one embodiment. The information processing system 100 includes a terminal 10 used by a user and an information processing device 20, and these terminals 10 and 20 are capable of communicating with each other via an arbitrary network NW such as the Internet and / or a local area network. The terminal 10 may be configured as any computer device such as a personal computer, smartphone, or tablet, and accepts text input from a user via any input device such as a keyboard, mouse, or touch panel, and transmits the text to the information processing device 20. The information processing device 20 generates a response text in response to the text accepted from the user and transmits it to the terminal 10. The terminal 10 then displays the response text on a display or other display device, allowing the user to know the content of the response to their input text.

[0016] The user can receive the desired response service by transmitting and receiving the text one or more times between the terminal 10 and the information processing device 20. On the terminal 10, any existing technology such as a web browser can be used to accept text input from the user in the response service and display the response results from the information processing device 20 to the user.

[0017] The operation settings and the like for the information processing device 20 can be performed by an operator or the like of a business providing a response service. After checking the response message to which the reference source information has been added and which has been automatically generated by the information processing device 20 according to this embodiment in response to a response request from a user, the operator can modify the response message and input it into the information processing device 20 if they determine that modification is necessary, and present it as the final response message to the user using the terminal 10.

[0018] The information processing device 20 may be configured as a single computer device as shown schematically in FIG. 2, or may be configured as two or more computer devices that can communicate with each other via a network NW, with the roles of each device being shared as described below.

[0019] As a variant, the terminal 10 in the configuration of Figure 2 may be omitted, and the user may directly operate an input device such as a keyboard provided on the information processing device 20 (e.g., a stand-alone computer device), thereby directly obtaining the response content on a display device such as a display provided on the information processing device 20.

[0020] As described above, text input from the user is input to the information processing device 20 by being accepted by the terminal 10 or by being accepted directly by the information processing device 20, and the text of the response content is also displayed on the terminal 10 or the information processing device 20, so that the user can know it. In the following explanation, reference to the points where input and output with the user takes place will be omitted, and the details of the processing by the information processing device 20 to respond to the user input will be explained.

[0021] 2 is a suitable example for implementing customer support or the like using the information processing system 100, but this embodiment is not limited to customer support applications and can be applied to any application for which a user requests a response by preparing a search database using the RAG method according to the application. For example, the information processing device 100 can also be used when a user wants a response that reflects knowledge, etc. based on search results from an internal document database, etc. using the RAG method. In this case, by adopting the above-described modified example, it is possible to use the information processing device 100 as if the user and the operator in FIG. 2 are the same person rather than different persons. In this case, a single user as the same person inputs text of a response request to the information processing device 100, and as a result, a response statement with reference source information added, etc., can be obtained as an output.

[0022] As described above, the information processing device 100 can automatically generate response messages for any purpose. In the following, the present embodiment will be described using the case where the information processing device 100 provides customer support as an example.

[0023] Figure 3 is a functional block diagram of an information processing device 20 according to one embodiment, and as shown in the figure, the device comprises a tentative response generation unit 1, a strategy identification unit 2, a document search unit 3, a document DB (database) 31, an information extraction unit 4, a response generation unit 5, a reference source linking unit 61, a reference source explanation unit 62, and a confirmation reception unit 63.

[0024] The tentative response generation unit 1, strategy identification unit 2, document search unit 3, information extraction unit 4, response generation unit 5, and reference source linking unit 61 all have an LLM unit 7 as their internal processing mechanism, which is trained in advance using large amounts of training data and is capable of general-purpose LLM processing. Figure 4 shows the processing of the LLM unit 7, which can accept input sentences containing questions or instructions and output output sentences that serve as answers. In addition, the LLM unit 7 can realize any text dialogue in the form of text input and output, not limited to the format of returning answers to questions, etc.

[0025] The input / output processing framework of this LLM unit 7 is the same as that of existing LLMs, and as is commonly used in the technical field, the input sentence and output sentence are also called input prompts and output prompts, respectively. Also, the configuration of the LLM unit 7 (the configuration when only the LLM unit 7 is extracted as a component of the information processing device 20) can be an LLM based on any existing learning model, similar to that used in the aforementioned Non-Patent Document 1, etc.

[0026] In this embodiment, the input sentences to the LLM unit 7 are prepared by the tentative response generation unit 1, strategy identification unit 2, document search unit 3, information extraction unit 4, response generation unit 5, and reference source linking unit 61, respectively, thereby freeing users and operators from the trouble of directly handling the LLM (for example, the trouble of inputting various input sentences by trial and error to obtain the desired answer and checking each time whether the answer is appropriate).

[0027] 5 is a flowchart of the operation of the information processing device 20 according to one embodiment, and the following describes each step of the flowchart while explaining the details of the processing content of each functional block of the information processing device 20. Note that in the functional block diagram of the information processing device 20 in FIG. 3, an outline of the data input and output between each functional block and between the functional block and a user or operator in the corresponding processing is indicated, and symbols representing the data are provided, such as symbols T0, T1, ST1, ST2, ST3, D1, D2, D3, R1, R2, R3, T2, T2R, and T2E.

[0028] 6 to 12 are diagrams showing various schematic examples for explaining the processing contents of the information processing device 20, and will be referred to sequentially in the following explanation. In these diagrams, the same symbols as those used to represent the data input and output in FIG. 3, such as symbol T0, are used, and specific examples of each data content are shown schematically in FIGS. 6 to 12.

[0029] ◆ Step S1...Temporary response generation unit 1 When the flow in Figure 5 starts, in step S1, the provisional response generation unit 1 accepts the text of a response request from the user, instructs the LLM unit 7 to generate text that responds to the response request, and outputs the response sentence (first response sentence) output by the LLM unit 7 to the next strategy identification unit 2, before proceeding to step S2.

[0030] 6 is a diagram showing a schematic example of the processing of the interim response generator 1. In this example, as described above, the user is a customer of a mobile phone or the like, and the user is requesting customer support in the information processing system 100. Specifically, the user is a parent who is seeking advice on a suitable smartphone for their child, and has entered response request text T0, "I would like to buy a smartphone for my child. Are there any advantageous plans?" As shown in example EX1, the interim response generator 1 creates a predetermined input prompt corresponding to the content of the response request text T0 by directly quoting the response request text T0, and instructs the LLM unit 7 to generate a response sentence in response to the response request T0, thereby obtaining a first response sentence T1 as an output prompt from the LLM unit 7.

[0031] In accepting the response request text T0, the type of the user's consultation content may be accepted in advance by selecting a menu on the browser, such as "new contract" or "fault consultation," and after specifying in advance that it corresponds to "new contract," the response request may be accepted as a specific direct input from the user in the form of text T0 "I want to buy a smartphone for my child. Are there any advantageous plans?". The input prompt example EX1 can be configured in this way by preparing predetermined instruction text OR1 for quoting the response request text T0 and generating a response based on rules for each individual consultation type, such as "new contract" or "fault consultation."

[0032] In constructing the predetermined instruction text OR1, as shown in example EX1, it is possible to include instructions for a predetermined role (such as a social role) as an LLM agent that the LLM unit 7 is expected to behave in its input and output, for example, "You are a customer support representative. Please generate a response to the customer request below."

[0033] The response sentence output by the provisional response generator 1 is referred to as the "first response sentence" to distinguish it from the "second response sentence," which is the response sentence output by the response generator 5 (described later). As can be understood from the above explanation, the first response sentence T1 is simply a response to the user's response request text T0, automatically generated as an output prompt by the LLM using only the standard input prompt EX1 based on the standard instruction text OR1. Therefore, if this first response sentence were to be used as the final response to the user, there is a considerable possibility that it would not be a satisfactory response, for example, because it contains hallucinations. Therefore, it is composed of the content of a "provisional response." The information processing device 20 further processes the first response sentence, which is the content of the provisional response, as described below to obtain a response that is expected to be revised to be more appropriate as a final response to the user, and can link the reference source information to the response sentence to reduce the effort required of an operator, etc.

[0034] In the above, the first response sentence as this "provisional response" is generated using only LLM, but in another embodiment, RAG may be used in addition to the LLM alone. In this case, the following processing should be performed in accordance with the RAG of the existing method. One or more documents containing content determined to be similar to the response request text T0 (i.e., documents that are expected to contain similar content and therefore provide a solution to the response request text T0) are searched for in the document DB 31. This search is performed by extracting features such as embedded expressions from the response request text T0, extracting similar features from each document in the document DB 31 in advance, and linking them together, and then finding documents whose features are determined to be similar based on a threshold value. By providing an input prompt to the LLM unit 7 indicating that a solution to the response request text T0 should be output as the first response sentence T1 and that the example document retrieved from the document DB 31 above should be considered as a reference case for creating this solution, the first response sentence T1 is obtained as an output prompt. (Note that in the example of Figure 6, the instruction to use the example document retrieved from the document DB 31 as a reference case is omitted.)

[0035] ◆ Step S2…Strategy Identification Unit 2 In step S2, the strategy identification unit 2 instructs the LLM unit 7 to identify the strategy for each text part of the first response sentence T1 obtained as described above, outputs the strategy list obtained from the LLM unit 7 to the document search unit 3, and outputs a log of this process to the reference source explanation unit 62 before proceeding to step S3.

[0036] Figure 7 is a diagram showing a schematic example of the processing of the strategy identification unit 2, and shows an example input prompt EX2 that the strategy identification unit 2 constructs to input to the LLM unit 7.The input prompt EX2 is composed of instruction text OR2 to divide the first response sentence T1 into parts (hereinafter, these divided ``parts'' may also be referred to as ``text parts'') and identify the strategy for each part, followed by a quote from the first response sentence T1 that is to be processed for strategy identification by the LLM unit 7.

[0037] The instruction text OR2 can be prepared as a standard phrase indicating that a strategy for each text portion should be identified from the first response text T1, such as the example "Below is the customer support response text. Please divide this response text into parts and identify a strategy for each part." If the provisional response text generation unit 1 knows the type of the user's consultation content, such as "new contract" or "breakdown consultation," by selecting from a menu or the like, the expression of the standard phrase may be prepared as an expression corresponding to the type of consultation content.

[0038] Although not included in the example sentence in Figure 7, instruction text OR2 may further include a phrase such as "a "strategy" is something that indicates, in the form of a heading, what the customer support representative is trying to achieve in the part of the response text that they are responding to," as a standard phrase for conveying to the LLM unit 7 a definition of the "strategy" that it is instructing the LLM unit 7 to identify. In addition, instruction text OR2 may further include sample cases in which strategies are actually identified for each part of the sample response text, thereby allowing the LLM unit 7 to more clearly understand the process of identifying strategies.

[0039] In Figure 7, the strategy identification unit 2 inputs the above input prompt EX2 into the LLM unit 7, and the resulting output prompt example EX21 is obtained by dividing the entire text of the first response sentence T1 into three text parts TP1, TP2, and TP3 in order, and identifying the strategies ST1, ST2, and ST3 for each of the text parts TP1, TP2, and TP3, as shown below. Text section TP1: "If your child is under XX years old, you can apply for the family discount. With the family discount, for example, you can save XX yen per month on a △△ contract." - Strategy ST1 in text section TP1... "Strategy: Overview of Family Discount" Text section TP2: "We also offer a service called 'Anshin Filter' to help you use the internet safely, so please consider using that as well." · Text section TP2 Strategy ST2… “Strategy: Information on additional services” Text section TP3: "This service is provided free of charge." · Text section TP3 Strategy ST3… “Strategy: Information on fees and application procedures for additional services”

[0040] In the example of Figure 7, the strategy identification unit 2 outputs the three strategies obtained as strategy lists ST1, ST2, and ST3 to the subsequent document search unit 3, so the three reference symbols "ST1, ST2, ST3" are also used in the functional block diagram of Figure 3, but generally a strategy list consisting of any n number of strategies (n≧1) is obtained. Similarly, the other reference symbols "D1, D2, D3" and "R1, R2, R2" in Figure 3 are shown as examples when there are three strategies in the strategy list, but this does not mean that there will always be three strategies; generally, there can be any n number (n≧1).

[0041] Furthermore, the strategy identification unit 2 outputs an operation log L2 indicating the processing content to the reference source explanation unit 62. The content of this log L2, as shown by the reference characters for the whole of FIG. 7, indicates that the output prompt EX21 was obtained by providing the input prompt EX2 to the LLM unit 7. That is, the log L2 is configured to include information on the input prompt EX2 and information on the output prompt EX21, as well as information on the processing step in which the output prompt EX21 was obtained from the input prompt EX2. (Note that the log L3 in FIG. 8 and the log L4 in FIG. 9, which will be described later, are also configured to include information on the input prompt and the output prompt, as well as information on the processing step in which the output prompt was obtained from the input prompt, in the same format as the log L2 in FIG. 7.)

[0042] In the above, the formal overall content of the logs L2, L3, and L4 was explained as including all information on the input / output process, but in each functional block that utilizes these logs L2, L3, and L4, the logs are utilized by extracting key points from them, as will be specifically explained at that time.

[0043] ◆ Step S3...Document Search Unit 3 In step S3, the document search unit 3 issues an input prompt to the LLM unit 7 instructing it to search the document DB 31 for documents containing relevant information for realizing each strategy in the strategy lists ST1, ST2, ST3 obtained from the strategy identification unit 2 as described above, thereby obtaining documents D1, D2, D3 containing relevant information for realizing each strategy ST1, ST2, ST3 as output prompts, and outputs these as pairs (D1, ST1), (D2, ST2), (D3, ST3) linked to the corresponding strategies to the downstream information extraction unit 4. In step S3, the document search unit 3 also outputs a log of the processing performed by the document search unit 3 to the referent linking unit 61 and the referent explanation unit 62, before proceeding to step S4.

[0044] To enable this search, the document DB 31 is constructed in advance as a collection of documents including solution examples, such as answers to inquiries that are desired to be realized by the information processing system 100. In the case of customer support, the document DB 31 is constructed in advance as a collection of documents recording the content of an inquiry from a user (customer) and how the operator's response to the content of the inquiry led to a solution, as well as manual documents such as an operation manual for a smartphone or the like.

[0045] In this example, documents D1, D2, and D3 are assumed to be retrieved for all strategies ST1, ST2, and ST3. However, there may be cases where retrieval is not possible due to circumstances such as the absence of appropriate documents in the document DB 31. If retrieval is not possible, the strategy in question can simply be discarded. For example, the strategy identification unit 2 obtains strategy ST4 in addition to the three strategies ST1, ST2, and ST3. If documents D1, D2, and D3 can be retrieved for the three strategies ST1, ST2, and ST3, respectively, but no relevant document can be retrieved for strategy ST4, strategy ST4 can be discarded, and only the three retrieved pairs (D1, ST1), (D2, ST2), and (D3, ST3) can be output to the information extraction unit 4 at the downstream stage.

[0046] Figure 8 shows a schematic example of the processing of the document search unit 3. In the document search unit 3, by providing the LLM unit 7 with an input prompt EX3 consisting of instruction text OR3, which is composed of a standard phrase to instruct the search for documents containing related information for realizing the strategies, ``Please search the database (document DB31) for documents that will realize each of the three strategies below,'' and a quote from the target strategy list ST1, ST2, ST3, the search results of documents D1, D2, D3 for all of the strategies ST1, ST2, ST3 can be obtained as output prompt EX31, and the strategy and document pairs (D1, ST1), (D2, ST2), (D3, ST3) can be output to the information extraction unit 4.

[0047] Note that, as a way of utilizing the LLM unit 7 to have the document search unit 3 search the document DB 31 for documents containing relevant information for realizing each strategy, similar to the applicant's Japanese Patent Application No. 2024-095526, "Dialogue Device, Program, and Method for Creating a Response Sentence to a User," the LLM unit 7 may generate a search query for searching the document DB 31 for the relevant document, and then perform a search from the document DB 31 using the search query. In other words, the input prompt for causing the LLM unit 7 to generate a search query may be an instruction to "generate a search query for document search that realizes the strategy from the strategy (and the corresponding text portion)." (Note that in the example of FIG. 8, such a query generation instruction and its results are omitted from the illustration.)

[0048] In addition, the document search unit 3 outputs its operation log L3 to the reference source linking unit 61 and the reference source explanation unit 62, and this log L3 is configured to include information on the input prompt EX3 and information on the output prompt EX31, as well as information on the processing process in which the output prompt EX31 was obtained from the input prompt EX3.

[0049] ◆ Step S4... Information Extraction Unit 4 In step S4, the information extraction unit 4 instructs the LLM unit 7 to itemize the related information R1, R2, R3 obtained from the document and its strategies (D1, ST1), (D2, ST2), (D3, ST3) obtained from the document search unit 3, and links the obtained related information to the corresponding strategy to obtain related information (ST1, R1), (ST2, R2), (ST3, R3) for each strategy, and outputs this to the response generation unit 5. In step S4, the information extraction unit 4 also outputs this operation log to the reference source linking unit 61 and the reference source explanation unit 62, and then proceeds to step S5.

[0050] That is, the information extraction unit 4 outputs its operation log L4 to the reference source linking unit 61 and the reference source explanation unit 62, and this log L4 is configured to include information on the input prompt EX4 and information on the output prompt EX41, as well as information on the processing process by which the output prompt EX41 was obtained from the input prompt EX4.

[0051] 9 is a diagram showing a schematic example of the processing of the information extraction unit 4. The input prompt to the LLM unit 7 is instruction text OR4, which can be configured as a fixed phrase: "Extract relevant information for realizing the corresponding strategy from each of the following three documents in itemized form." The input prompt EX4 includes the documents to be processed according to this instruction and their strategies (D1, ST1), (D2, ST2), and (D3, ST3). The search results are output as output prompt EX41. That is, relevant information R1 for realizing strategy ST1 is obtained in itemized form from document D1, relevant information R2 for realizing strategy ST2 is obtained in itemized form from document D2, and relevant information R3 for realizing strategy ST3 is obtained in itemized form from document D3. Based on these results, the relevant information for each strategy (ST1, R1), (ST2, R2), and (ST3, R3) can be output to the response generation unit 5.

[0052] In addition to linking reference sources, the following secondary effects can be expected from this embodiment: In other words, the information extraction unit 4, which has undergone a series of processes on the first response sentence generated by the provisional response generation unit 1 and which may contain problems such as hallucination, extracts related information from documents searched in the document DB 31, which is expected to store fact-based documents such as internal company documents.This related information is expected to be fact-based, and in the second response sentence generated by the response generation unit 5 at the subsequent stage based on this, it can be expected that the hallucination in the first response sentence will have been eliminated or reduced.

[0053] Step S5...Response Generation Unit 5 In step S5, the response generation unit 5 instructs the LLM unit 7 to generate a user response sentence by realizing each strategy using the related information corresponding to the related information (ST1, R1), (ST2, R2), and (ST3, R3) for each strategy obtained by the information extraction unit 4, thereby obtaining a second response sentence T2 and outputting this second response sentence T2 to the reference source linking unit 61. This instruction may include an instruction to the effect that the response request text T0 is a user request, citing the response request text T0 as the original content for which a user response is requested (although this is not shown in the example of FIG. 10 described later). It should be noted that the relevant information for each strategy (ST1, R1), (ST2, R2), and (ST3, R3) in their current form is considered to be somewhat sufficient as a response to a user request, but since it is an information list, it is considered inappropriate to use it as a response to the user in its current form, so it is preferable to convert it into the format of a response statement.In addition, the information list may include information that is unnecessary for responding to the user, and the response generation unit 5 can generate a response statement from which unnecessary information has been deleted.

[0054] Step S61...Reference source linking unit 61 In step S61, the reference source linking unit 61 refers to the log L3 obtained by the document search unit 3 and the log L4 obtained by the information extraction unit 4, and instructs the LLM unit 7 to link the information of the reference source of the relevant part to the relevant part in the document for the second response sentence T2 obtained by the response generation unit 5 described above, and outputs the second response sentence T2R linked to the reference source obtained as an output prompt to the reference source explanation unit 62, and then proceeds to step S62.

[0055] Specifically, the reference source linking unit 61 can obtain the second response sentence T2R linked to the reference source as an output prompt by configuring the input prompt to the LLM unit 7 as follows: That is, the input prompt can be configured as an instruction to add the reference source to the second response sentence T2 obtained by the response generation unit 5 based on the itemized extraction results R1, R2, and R3 in the output of log L4 obtained by the information extraction unit 4 and the information on the reference source documents D1, D2, and D3 in log L3 obtained by the document search unit 3. That is, by searching for the relevant portion P1 of the extraction result R1 in the second response message T2 (i.e., the relevant portion P1 in the second response message T2 where the content equivalent to the extraction result R1 is written) and the relevant portion d1 of the referenced document D1 of the extraction result R1 (i.e., the relevant portion d1 of the referenced document D1 where the content equivalent to the extraction result R1 is written), and instructing the LLM unit 7 to output the results, the information to add the reference source can be obtained as an output prompt with the following content, and can be linked as the reference source information. The partial statement P1 in the second response message T2 is referenced from the partial statement d1 in the document D1.

[0056] In addition to the above, the input prompt for providing linking may be configured in the manner of an example described below. Also, at the information extraction unit 4 stage, when extracting the extraction result R1, the location d1 corresponding to R1 may be simultaneously output to the LLM unit 7, and the reference source linking unit may search only for the corresponding location P1 of the extraction result R1, thereby enabling linking to location d1.

[0057] 10 is a diagram showing a schematic example of the processing of the response generation unit 5 and the reference source linking unit 6. In the response generation unit 5, the input prompt EX5 can be composed of a fixed phrase, and by providing the LLM unit 7 with instruction text OR5, which indicates that a response should be generated from related information corresponding to the strategies, "Please generate a customer support response from the following three strategies and the related information that realizes them," and citations (ST1, R1), (ST2, R2), and (ST3, R3) of the data to be processed by this instruction, the second response sentence T2 can be obtained as its output prompt.

[0058] 10 further shows an example of a second response text T2R obtained by linking a reference source to the second response text T2 as a schematic example of the processing by the reference source linking unit 6. In this reference source linked second response text T2R, the reference source information Ref1, Ref2, Ref3, and Ref4 are linked to each of four text portions P1, P2, P3, and P4 that appear in order in the original second response text T2. Of the four pieces of reference source information, Ref2 is marked as "reference source unknown" and the reference source could not be found, but the remaining three, Ref1, Ref3, and Ref4, are information that identifies the reference source by URL or the like. As will be described later with reference to the example in Figure 1 again, and as already explained as "●The reference source for partial description P1 in the second response text T2 is partial description d1 in document D1," the reference source linking unit 6 can obtain the linking of the reference source, including information on which specific part (part d1, etc.) in the reference source document (document D1, etc.) constitutes the reference source.

[0059] FIG. 11 shows an example of a prompt used by the reference source linking unit 6 to generate the second response text T2R obtained by linking the reference sources as described above. The general configuration of the input prompt is as described above. By citing the itemized extraction result R1 and its corresponding source document D1 and second response text T2, and searching for the corresponding section P1 of extraction result R1 in second response text T2 and the corresponding section d1 of extraction result R1 in document D1, the reference source can be output, assuming that the reference source of section P1 in second response text T2 is section d1 in document D1. The same applies to obtaining the corresponding sections d2 and d3 that are the reference sources of extraction results R2 and R3, respectively. In the specific example of FIG. 11, because the instructions (tasks) to the LLM unit 7 are somewhat complex, examples of actual extractions of references using similar tasks are provided as references, and the actual tasks to be performed are also provided. The upper part A11 is the instruction given as a reference, and the lower part B11 is the task actually given.

[0060] The sample instructions in the upper portion A11 are as follows: portion A11a is a task explanation, and portion A11b shows the specific task content together with the following citation. A11a: You are responsible for verifying the references of information contained in answers generated by AI. A11b… “You will be given the presented response (→ second response T2) and information about the references that may have been given to the AI ​​to generate that response (→ document and extracted content pairs, (D1, R1), (D2, R2), (D3, R3)). Please add the references that you think the AI ​​actually used to generate the response for each part.”

[0061] As shown in parts A11a and A11b, the second response text T2 (not a sample instruction but an actual instruction) is AI-generated (generated by the LLM unit 7), and the documents and their extracted contents (D1, R1), (D2, R2), and (D3, R3) may have been used as references at that time. Information regarding the generation process of each piece of information T2, (D1, R1), (D2, R2), and (D3, R3) may be included in the input prompt by the reference source linking unit 6.

[0062] Furthermore, as an actual instruction, the input prompt in the reference source linking unit 6 may include the following, as exemplified in the lower portion B11 showing a part of the cited data to be searched excluding the cited data. ●However, if you cannot find an appropriate reference source for the information, please write (Reference: Unknown). For statements that do not require any evidence, such as "Please let us know if you have any questions," you do not need to add anything.

[0063] In other words, when the LLM unit 7 searches for the parts of the second response text T2 that correspond to the document and its extracted content (D1, R1), (D2, R2), (D3, R3), and when there is no part in the second response text T2 that corresponds to the document and its extracted content (D1, R1), (D2, R2), (D3, R3) as a reference source for one or more parts of the second response text T2, and the part is other than the standard phrase such as "Please let us know if you have any questions," and corresponds to a fact that is the subject of the judgment of authenticity, the reference source linking unit 6 may include an instruction in the input prompt to output "reference source unknown."

[0064] 10 illustrates that the reference source Ref1 of the text portion P1 is indicated in the second response message T2, but the reference source linking unit 6 of this embodiment can also obtain information on which specific part of the document that constitutes this reference source Ref1 is the reference source, as shown in the example of Fig. 1. That is, if the reference source Ref1 is the entire text T206 in Fig. 1, this embodiment can also obtain information that a specific text portion T207 within the entire text T206 is the reference source.

[0065] ◆ Step S62...Reference source commentary section 62 In step S62, the reference source linking unit 62 links an explanation to each of the reference sources in the second response sentence T2R to which the reference source obtained by the reference source linking unit 61 is linked, and outputs the second response sentence T2E in which the reference source and its explanation are linked to the confirmation receiving unit 63, and then proceeds to step S63.

[0066] By referring to the logs L2, L3, and L4, the reference source explanation unit 62 can obtain an explanation of each reference source in the second response text T2R, i.e., information on what role this reference source is expected to play in generating response texts for customer support, etc. (information on the reason for being referenced). Specifically, for each reference source in the second response text T2R linked to the reference source output by the reference source linking unit 61, all or part of the contents listed below can be shown as details. (1) The corresponding strategy is identified from the log L2 output by the strategy identification unit 2, that is, information on the purpose for which the referencing document was used. (2) Information on the referencing document (such as information that can identify the document, such as a URL) is shown from the log L3 output by the document search unit 3. (3) The log L4 output by the information extraction unit 4 lists related information (corresponding to the strategy) in the referenced document, that is, information on what information in the referenced document was used.

[0067] As mentioned above, the reference source linking unit 61 obtains the reference source information in the form of "● For the partial statement P1 in the second response text T2, its reference source is the partial statement d1 in document D1.", and therefore, the above information (1) to (3) can be obtained. That is, (1) can be obtained as the strategy ST1 to which document D1 corresponds, (2) can be obtained as document D1, and (3) can be obtained as related information R1 to which document D1 corresponds, and by obtaining the correspondence from each log, the information (1) to (3) can be obtained.

[0068] Figure 12 is a diagram showing a schematic example in which further explanations are linked to reference sources in the reference source explanation unit 62, where reference source Ref1 is linked to its explanation Exp1, and reference source Ref2 is linked to its explanation Exp2, and the content of the explanation is information that explains why the reference source is referenced by showing specifically what "the information obtained from the reference source text and its purpose" is.

[0069] Step S63...Confirmation reception unit 63 In step S63, the confirmation receiving unit 63 displays the second response text T2E, which links the reference source and its explanation obtained by the reference source explanation unit 62, to the operator, so that the operator can confirm its content, and if the operator determines that revisions or other refinements are necessary, the operator accepts the refinements and then outputs the second response text to the user by sending it to the user, thereby completing the flow of Fig. 5. For example, if a search of the document DB 31 reveals that the second response text T2E contains information that is private information of someone unrelated to the user, it is possible to delete the unnecessary information and provide the final response text to the user.

[0070] As described above, according to this embodiment, the operator can check not just the response text but the response text that is linked to the reference source and explanation for each text portion at the confirmation receiving unit 63, so that various contents can be confirmed smoothly and confirmation costs can be reduced.

[0071] Various supplementary and alternative examples will be explained below.

[0072] (1) The embodiments of the present invention can provide greater convenience for RAG, which has been actively researched and developed recently as a powerful application of LLM, a type of generative AI (artificial intelligence) that automatically generates text. This can contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote inclusive and sustainable industrialization, and promote industrialization and inclusive technology."

[0073] (2) The referrer linked by the referrer linking unit 61 may be highlighted, for example, so that an operator can confirm it. In the example of text T206 in Figure 1, the text portion T207 of the referrer may be processed, for example, by inverting the black and white of the referrer text. When a user accesses the referrer document, the highlighting may be added to the original text. Any existing method can be used for this implementation. For example, if the processing is performed on the web page side, the query parameter "highlight" may be received (e.g., https: / / example.com / page.html?highlight=) and search processing and highlighting may be performed. If the processing is performed on the system side, a relay page may be prepared by a proxy and the accessed page may be displayed on it using an iframe or the like. Search processing and highlighting may be performed based on the original text information.

[0074] (3) When obtaining related information, the information extraction unit 4 instructs the LLM unit 7 to obtain it in itemized form, but this instruction to obtain itemized information may be omitted. However, if the instruction is not omitted, the following advantages can be obtained. ●When a document contains a large amount of information, depending on the performance of the search engine, it may find documents that are not particularly relevant. By inserting a task such as itemizing the necessary information, it is possible to determine when there is no information to extract, preventing unnecessary documents from being passed on to subsequent stages. ●If you extract only the main points in bullet points beforehand, your answers will be more concise, and you will be able to narrow down the points you link as reference sources to just the main points. ● Information extraction can be said to be similar to summarization tasks, but LLM inherently shows high performance in summarization tasks, so hallucination is a very low process. In other words, it is basically possible to check bullet points without checking the entire page.

[0075] (4) In this embodiment, after the response generation unit 5 generates the second response sentence, the reference source is further linked by the reference source linking unit 61, and instructions are given to the LLM unit 7 in two separate steps. This could be done all at once, i.e., a single instruction to the LLM unit 7 could be used to generate a response sentence in a format that links the reference source, but in this embodiment, the process is divided into two steps, which offers the following advantages. ●If the source of the reference is also included when generating an answer, LLMs will tend to simply copy the information given in the RAG and will not make use of their own knowledge, so this should be avoided. ●First, by allowing the user to generate answers with a certain degree of freedom, they can flexibly integrate information, reinforce it with their own knowledge, and then add reference sources, thereby obtaining higher quality answers with reference source information.

[0076] (5) There may be cases where the referrer linking unit 61 is unable to link the referrer, resulting in a "reference source unknown" status. This is most likely due to one of the following two reasons. ●This is the part of the text that corresponds to general information and common sense that LLM students have acquired through prior learning. ●This is the part of the text that corresponds to information that could not be obtained from the manual but that absolutely had to be mentioned in order to make the answer valid (to satisfy the user).

[0077] (6) The latter of the two types above is intended to encourage further enrichment of the data in the document DB 31 in order to prevent such situations. Therefore, if the reference source becomes “unknown,” a notification that the reference source is “unknown” may be sent to the operator, etc., indicating that caution may be required (without separately analyzing which of the two types above is actually the cause).

[0078] (7) FIG. 13 is a diagram showing an example of the hardware configuration of a general computer device 70. The terminal 10 and the information processing device 20 in the information processing system 100 can each be realized as one or more computer devices 70 having such a configuration. When the terminal 10 or the information processing device 20 is realized using two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, one or more dedicated processors 72, such as a GPU (Graphics Processing Unit) and / or an NPU (Neural Network Processing Unit), that execute some or all of the CPU 71's execution instructions in place of or in cooperation with the CPU 71 and are specialized for specific operations, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the dedicated processor 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76 that provides display output, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, or the like, a speaker 78 that outputs audio, a microphone 79 that inputs audio, and a bus BS for transmitting and receiving data among these.

[0079] Each functional unit of the information processing device 20 can be realized by a CPU 71 and / or a dedicated processor 72 that reads from a ROM 74 and executes a predetermined program corresponding to the function of each unit. Both the CPU 71 and the dedicated processor 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and dedicated processor 72. When communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and dedicated processor 72. When audio input / output-related processing is performed, a speaker 78 and a microphone 79 also operate in conjunction with the CPU 71 and dedicated processor 72.

[0080] When inputting and outputting text data and voice by a user or operator in the terminal 10 or the information processing device 20, a display 76, an input interface 77, a speaker 78, a microphone 79, etc. may be used. Voice input data may be converted into text using existing voice analysis technology, and text input may be accepted. [Explanation of symbols]

[0081] 100...information processing system, 10...terminal, 20...information processing device, 1...provisional response generation unit, 2...strategy identification unit, 3...document search unit, 31...document DB, 4...information extraction unit, 5...response generation unit, 61...reference source linking unit, 62...reference source explanation unit, 63...confirmation reception unit, 7...LLM unit

Claims

1. a first process of generating a first response sentence in response to a response request using a large-scale language model; a second process of identifying a response strategy for each text portion of the first response sentence using the large-scale language model; a third process of searching a database for each of the strategies using the large-scale language model to retrieve documents for implementing the strategy; a fourth process of extracting relevant information for implementing a corresponding strategy from each of the documents using the large-scale language model; a fifth process of generating a second response sentence in response to the response request from each of the strategies and associated information corresponding to the strategies using the large-scale language model; and a sixth process of obtaining reference source information for each text portion of the second response sentence by referring to logs of the third process and the fourth process using the large-scale language model.

2. The information processing device described in claim 1, characterized in that in the sixth process, information on the reference source for each text portion of the second response message is obtained, including identification of the document searched from the database that is the reference source and identification of the referenced portion within the document.

3. The information processing device according to claim 1, characterized in that in the sixth process, the input to the large-scale language model includes an instruction to search the second response sentence for a portion corresponding to the document and related information extracted from the document, thereby referencing the logs of the third process and the fourth process.

4. The information processing device described in claim 1 further executes a seventh process to obtain explanatory information regarding the reference source for each text portion of the second response message obtained in the sixth process by referring to at least one log of the second process, the third process, and the fourth process.

5. 5. The information processing apparatus according to claim 4, wherein in the seventh process, a log of the second process is referenced to obtain information on a strategy corresponding to a reference source as information on an explanation regarding the reference.

6. 5. The information processing apparatus according to claim 4, wherein the seventh process refers to the log of the third process to obtain information for identifying a document corresponding to a reference source as information for explaining the reference.

7. The information processing device according to claim 4, characterized in that in the seventh process, the log of the fourth process is referenced to obtain information used in the document corresponding to the reference source as information on an explanation regarding the reference.

8. 2. The information processing apparatus according to claim 1, wherein the sixth process further includes a notification that no reference source association has been obtained for a text portion for which no reference source association has been obtained.

9. a first process of generating a first response sentence in response to a response request using a large-scale language model; a second process of identifying a response strategy for each text portion of the first response sentence using the large-scale language model; a third process of searching a database for each of the strategies using the large-scale language model to retrieve documents for implementing the strategy; a fourth process of extracting relevant information for implementing a corresponding strategy from each of the documents using the large-scale language model; a fifth process of generating a second response sentence in response to the response request from each of the strategies and associated information corresponding to the strategies using the large-scale language model; and a sixth process of obtaining reference source information for each text portion of the second response sentence by referring to the logs of the third process and the fourth process using the large-scale language model.

10. a first process of generating a first response sentence in response to a response request using a large-scale language model; a second process of identifying a response strategy for each text portion of the first response sentence using the large-scale language model; a third process of searching a database for each of the strategies using the large-scale language model to retrieve documents for implementing the strategy; a fourth process of extracting relevant information for implementing a corresponding strategy from each of the documents using the large-scale language model; a fifth process of generating a second response sentence in response to the response request from each of the strategies and associated information corresponding to the strategies using the large-scale language model; and a sixth process of obtaining reference source information for each text portion of the second response sentence by referring to the logs of the third process and the fourth process using the large-scale language model.

Images