Information processing device, method, and program

Abstract

An information processing device capable of generating an appropriate response message is provided. [Solution] The system executes the following steps in response to a response request from a user: a first process 1, which searches for similar documents containing descriptions of similar requests from a first database storing documents of solved cases; a second process 2, which uses a large-scale language model to generate a first response statement from the similar documents and the response request statement; a third process 3, which uses the large-scale language model to assign strategies to each part of the response request statement; a fourth process 4, which uses the large-scale language model to search a second database storing documents containing descriptions of facts for a verification source document to verify the facts for each part to which the strategy has been assigned; and a fifth process 5, which uses the large-scale language model to generate a second response statement for the response request statement from the verification source document and the first response statement to which a strategy has been assigned for each part.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, method, and program for performing processing using a large-scale language model (LLM). [Background technology]

[0002] Companies are facing the challenge of increasing labor costs for operators and employees who handle customer support and internal inquiries. QA (question and answer)-based chatbots have been introduced, but many of the existing chatbots only respond based on pre-prepared QA scenarios, and unexpected questions require individual responses by humans, which limits the reduction in labor costs.

[0003] Meanwhile, large-scale language models (hereafter referred to as LLMs), which have emerged and developed in recent years as generative AI for text, are expected to be used to generate responses similar to those given by operators to such unexpected questions, and to provide systems that support operators in creating responses. LLMs are not limited to customer support and internal inquiries within companies, but can also be used to automatically generate responses of any content, such as product recommendations. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] GAO, Yunfan, et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023, 2. Summary of the Invention [Problem to be solved by the invention]

[0005] An existing approach for generating responses and other correspondence using LLM is a method called Search Augmentation Generation (hereinafter abbreviated as RAG), which is disclosed in, for example, Non-Patent Document 1. LLM is trained using a huge amount of data and is highly versatile. However, applying LLM directly to a specific field (e.g., customer support, internal inquiries, etc.) where the vast amount of data from which it is trained does not sufficiently contain the relevant knowledge may not necessarily produce appropriate results. A known problem with LLM is that false answers may be obtained as a result of so-called hallucination. However, the RAG method makes it possible to simply specialize LLM for a specific field. However, even in an LLM specialized for a specific field using the RAG method, hallucination still occurs, and various methods exist to counter this.

[0006] On the other hand, with conventional technology, even though the response did not involve hallucination, it could still be inappropriate because the search results were reflected from a database that recorded actual cases in the form of past logs, etc. Figure 1 is a diagram that schematically illustrates the issues with this conventional technology, using an example of an attempt to automate customer support using the RAG method. Because individual cases, such as examples, are so diverse, it is not always possible to find a case in the database that perfectly matches the case being inquired about. Similarly, the cases searched from the database may contain unnecessary and / or inconsistent information for the case being inquired about, resulting in an inappropriate response that is, so to speak, localized.

[0007] That is, Example EX2 is the result of searching for past cases similar to the user inquiry shown in Example EX1, "Q: I want to buy a smartphone for my child. Are there any good plans?", and a document is retrieved that contains a specific user inquiry (Q) and its answer (A) as a set, recorded as text content: "Q: What plan is good if I'm buying a smartphone for an 8-year-old child? A: Plan A is good for an 8-year-old child." The answer to the user inquiry is automatically generated based on the document in the search results, and is shown in Example EX3, "A: Plan A is good for an 8-year-old child."

[0008] However, while this example answer EX3 does not constitute hallucination because it assumes an "8-year-old child" based on past cases, if the actual age of the inquiring user's child were 18, not 8, and Plan A could not be applied, this automatically generated answer could have been inappropriate with prior art. Furthermore, because the content simply searched from the database for such information as "8-year-old child" could be considered personal information in past cases, the answer contains fragmented personal information. While this does not constitute a direct leak of personal information, the indirect mention of unrelated personal information could have resulted in an inappropriate answer with prior art. On the other hand, excessive masking of personal information in the database content could prevent indirect leaks of personal information, but could also reduce the accuracy of the information extracted from the database, resulting in a trade-off between personal information and answer accuracy with prior art. In the above explanation, the term "leakage" is used in relation to "personal information, etc." to make it easier to understand, but as is clear from the context, this corresponds to "fragmentary reference to personal information, etc." and has not previously occurred as a breach of personal information protection. (We use a database that has undergone minimal masking to prevent such breaches from occurring.)

[0009] In view of the above-mentioned problems with the conventional technology, an object of the present invention is to provide an information processing device, method, and program that can generate an appropriate response message. [Means for solving the problem]

[0010] To achieve the above object, the present invention is an information processing device that executes the following steps: a first process of searching a first database storing documents of solution cases for similar documents containing descriptions of similar requests in response to a response request statement from a user; a second process of using a large-scale language model to generate a first response statement from the similar documents and the response request statement; a third process of using the large-scale language model to assign strategies to each part of the response request statement; a fourth process of using the large-scale language model to search a second database storing documents containing descriptions of facts for a verification source document to verify each part to which the strategies are assigned; and a fifth process of using the large-scale language model to generate a second response statement in response to the response request statement from the verification source document and the first response statement to which the strategies are assigned for each part. The present invention is also characterized by a method and a program corresponding to the device. [Effects of the Invention]

[0011] According to the present invention, by performing the third, fourth, and fifth processes to abstract the response message from the second process and then generating the response message, it is possible to generate an appropriate response message. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram schematically illustrating a problem with the conventional technology. [Figure 2] 1 is a configuration diagram of an information processing system according to an embodiment. [Figure 3] FIG. 1 is a functional block diagram of an information processing device according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating the processing of the LLM unit. [Figure 5] 1A and 1B are diagrams illustrating examples for schematically and generally explaining the effects of the present embodiment. [Figure 6] 10 is a flowchart illustrating an operation of an information processing device according to an embodiment. [Figure 7] FIG. 10 is a diagram showing a schematic example of an input / output prompt in the first generating unit. [Figure 8] FIG. 10 is a diagram showing a schematic example of an input / output prompt in the strategy decomposition unit. [Figure 9] FIG. 10 is a diagram showing a schematic example of an input / output prompt when fact verification is performed as additional processing in the strategy decomposition unit. [Figure 10] 10A and 10B are diagrams showing schematic examples of input / output prompts and search results in a second search unit. [Figure 11] FIG. 10 is a diagram showing a schematic example of an input / output prompt in a second generation unit. [Figure 12] FIG. 1 is a diagram illustrating a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE INVENTION

[0013] 2 is a configuration diagram of an information processing system according to one embodiment. The information processing system 100 includes a terminal 10 used by a user and an information processing device 20, and these terminals 10 and 20 are capable of communicating with each other via an arbitrary network NW such as the Internet and / or a local area network. The terminal 10 may be configured as any computer device such as a personal computer, smartphone, or tablet, and accepts text input from a user as a response request via any input device such as a keyboard, mouse, or touch panel, and transmits the text to the information processing device 20. The information processing device 20 generates a response text in response to the text accepted from the user and transmits it to the terminal 10. The terminal 10 then displays the response text on a display or other display device, allowing the user to know the content of the response to their input text.

[0014] The user can receive the desired response service by transmitting and receiving the text between the terminal 10 and the information processing device 20. On the terminal 10, any existing technology such as a web browser can be used to accept text input from the user in the response service and display the response results from the information processing device 20 to the user.

[0015] An operator of a business providing a response service can set up the operation of the information processing device 20. After checking the response message that the information processing device 20 automatically generates in accordance with this embodiment in response to a response request from a user, the operator can modify the response message, if necessary, and then input it into the information processing device 20 to present it as the final response message to the user using the terminal 10.

[0016] The information processing device 20 may be configured as a single computer device as shown schematically in FIG. 2, or may be configured as two or more computer devices that can communicate with each other via a network NW, with the roles of each device being shared as described below.

[0017] As a variant, the terminal 10 in the configuration of Figure 2 may be omitted, and the user may directly operate an input device such as a keyboard provided on the information processing device 20 (e.g., a stand-alone computer device), thereby directly obtaining the response content on a display device such as a display provided on the information processing device 20.

[0018] As described above, text input from the user is input to the information processing device 20 by being accepted by the terminal 10 or by being accepted directly by the information processing device 20, and the text of the response content is also displayed on the terminal 10 or the information processing device 20, so that the user can know it. In the following explanation, reference to the points where input and output with the user takes place will be omitted, and the details of the processing by the information processing device 20 to respond to the user input will be explained.

[0019] 2 is a suitable example for implementing customer support etc. using the information processing system 100, but this embodiment is not limited to customer support applications and can be applied to any application for which a user requests a response by preparing a search database using the RAG method according to the application. As described above, the information processing system 100 can realize the automatic generation of response messages for any application, but the following description of this embodiment will be given taking the case of implementing customer support using the information processing system 100 as an example.

[0020] Figure 3 is a functional block diagram of an information processing device 20 according to one embodiment, and as shown, the information processing device 20 includes a first search unit 1, a first generation unit 2, a strategy decomposition unit 3, a second search unit 4, a second generation unit 5, a first DB (database) 11, and a second DB 41.

[0021] The first generation unit 2, strategy decomposition unit 3, second search unit 4, and second generation unit 5 all have an LLM unit 7 as their internal processing mechanism, which is trained in advance using large amounts of training data and is capable of general-purpose LLM processing. Figure 4 shows the processing of the LLM unit 7, which can accept input sentences containing questions, instructions, etc., and output output sentences that serve as answers. In addition, the LLM unit 7 can realize any text dialogue in the form of text input / output, not limited to the format of returning answers to questions, etc.

[0022] The input / output processing framework of this LLM unit 7 is the same as that of existing LLMs, and as is commonly used in the technical field, the input sentence and output sentence are also called input prompts and output prompts, respectively. Also, the configuration of the LLM unit 7 (the configuration when only the LLM unit 7 is extracted as a component of the information processing device 20) can be an LLM based on any existing learning model, similar to that used in the aforementioned Non-Patent Document 1, etc.

[0023] In this embodiment, the input sentences to the LLM unit 7 are prepared by the first generation unit 2, the strategic decomposition unit 3, the second search unit 4, and the second generation unit 5, respectively, thereby freeing users and operators from the trouble of directly handling the LLM (for example, the trouble of inputting various input sentences by trial and error to obtain the desired answer and checking each time whether the answer is appropriate).

[0024] FIG. 5 is a diagram showing examples for schematically and broadly explaining the effects of this embodiment. Examples EX1-EX3 in the first stage are identical to the conventional example in FIG. 1. In the information processing device 20 of this embodiment, as shown in Examples EX1-EX3, the first search unit 1 and first generation unit 2 in the first stage generate answer sentences as if they were provisional, as in the conventional RAG method (however, in this embodiment, the first DB 11 is particularly used as the database for search in RAG). Further, the strategy decomposition unit 3, second search unit 4, and second generation unit 5 in the second stage perform additional processing on these provisional answer sentences, resulting in strategy decomposition, search, and final answer generation processes as shown in Examples EX4, EX5, and EX6, respectively. As a result, whereas the conventional answer example EX3 in FIG. 1 was inappropriate because it assumed the child's age to be 8, this embodiment can obtain an appropriate planned answer appropriate to the child's age, as shown in Final Answer Example EX6.

[0025] FIG. 6 is a flowchart of the operation of the information processing device 20 according to one embodiment, and the details of the processing content of each functional block of the information processing device 20 will be described below while explaining each step of the flowchart.

[0026] Step S1...First search unit 1 (and first DB 11) In step S1, a response request statement (response request text) is received from the user, documents containing content similar to this response request statement are searched for in the first DB 11, and the searched similar documents are output to the first generation unit 2, after which the process proceeds to step S2. As shown in FIG. 3, this response request text received as input by the first search unit 1 is also input to the first generation unit 2, strategy decomposition unit 3, second search unit 4, and second generation unit 5. Note that the "response request statement" will be explained using an example of an "inquiry (statement, content)," which is expected to be a common specific content in the case of customer support. In this case, if it is not limited to customer support, it will be referred to as "general," etc.

[0027] The first search unit 1 can search for similar documents from the first DB 11 by the following processes p1 to p4. ●Process p1: The content of the user's inquiry is converted into an embedding representation, which is an existing method, as a feature representation of the text. Process p2: Convert the query content in each document stored in the first DB 11 into an embedded representation. Process p3: Calculate the similarity between the embedded expression of the user's inquiry and the embedded expression of each document in the first DB 11. Process p4: Documents in the first DB 11 that are highly similar to the inquiry content are retrieved as search results.

[0028] In the first DB 11, multiple (N) documents D(i) (i = 1, 2, ..., N) containing user inquiries and their corresponding solutions are stored, along with the contents of the inquiry Q(i) (i = 1, 2, ..., N) and the solutions A(i) (i = 1, 2, ..., N). Furthermore, the embedded representation emb(i) (i = 1, 2, ..., N) of the inquiry Q(i) may also be extracted in advance and associated with D(i) = (Q(i), A(i), emb(i)). This eliminates the need for conversion each time in step p2. In step p4, a single document with the highest similarity may be obtained, or one or more documents whose similarity exceeds a threshold may be obtained. This first DB 11 may be prepared in advance, recording various specific examples of past customer support.

[0029] Note that example EX1 in Figure 5 (and Figure 1) mentioned above is an example of a query sentence from a user, and example EX2 is an example of a document searched by the first search unit 1 from the first DB11 as a document containing a query similar to this query.

[0030] ◆ Step S2... First generation unit 2 In step S2, the first generation unit 2 inputs to the LLL unit 7 as an input prompt an instruction to generate an answer sentence (generally a response sentence) to the user inquiry using the similar documents as reference examples for specific solutions (generally responses), citing the inquiry content input by the user in step S1 and similar documents searched for as containing inquiries similar to the inquiry content, obtains the answer sentence as an output prompt, outputs this answer sentence to the strategy decomposition unit 3, and then proceeds to step S3.

[0031] 7 shows a schematic example EX20 of an input / output prompt in the first generating unit 2. Note that the output example of example EX20 is the same as example EX3 in FIG. 5 (and FIG. 1) described above.

[0032] ◆ Step S3…Strategy decomposition part 3 In step S3, the strategy decomposition unit 3 inputs to the LLM unit 7 an instruction to output strategies for each part of the inquiry content and the response text as an input prompt citing the inquiry content from the user in step S1 (generally a response request text) and the response text (generally a response text) in step S2 (as a provisional response), and obtains the result of linking the strategies to each part of the response text as an output prompt (generally a response text with a strategy assigned to each part), outputs the result to the second search unit 4 and the second generation unit 5, and then proceeds to step S4.

[0033] FIG. 8 shows a schematic example EX30 of an input / output prompt in the strategy decomposition unit 3. In this schematic example EX30, specific examples of actually extracted strategies are also included in the input / output prompt so that the instruction "to output strategies for each part of the response sentence" can be more clearly understood by the LLM unit 7. Furthermore, the input prompt may be configured to provide a definition of "strategy" such that the agent in the agent utterance (the response sentence automatically generated by the LLM) has a personality, and that the strategy corresponds to the intention or aim of the part of the utterance. (Note that in example EX30 of FIG. 8, the definition of "strategy" is not explicitly provided, as it is assumed that the definition can be understood from the context of the input prompt.)

[0034] In the output example of Example EX30 in Figure 8, the entire answer sentence "Plan A would be a good deal for an 8-year-old child" is divided into the following two parts, and the strategies are obtained as follows. Part 1: "For an 8-year-old child" → Strategy for Part 1: "Check the child's age" Part 2: "Plan A is a good deal" → Strategy for Part 2: "Offer a plan that suits the child's age"

[0035] ◆ Additional processing in step S3 In step S3, the strategy decomposition unit 3 may use the answer sentences to which strategies have been assigned for each part of the output sentence obtained from the LLM unit 7 as described above to give further instructions to the LLM unit 7. Specifically, each part to which a strategy has been assigned is quoted and input to the LLM unit 7 as an input prompt instructing the LLM unit 7 to determine whether or not the content should be fact-checked, and a result as an output prompt indicating whether or not each part should be subject to fact-checking is obtained. The additional results obtained are also output to the second search unit 4.

[0036] Fig. 9 shows a schematic example EX31 of an input / output prompt when fact verification is performed as an additional process in the strategy decomposition unit 3. In this example EX31, an actual case of fact verification (fact checking) is included in the input prompt as a reference, and the necessity of fact verification for the first part "For an 8-year-old child" is obtained as "No. Reason: This is a confirmation of the child's attributes, and there is no need to search." (Although omitted in Fig. 9, the necessity of fact verification for the second part is obtained as "Needed," for example.)

[0037] Step S4... Second search unit 4 (and second DB 41) In step S4, the second search unit 4 performs the following steps: (1) generating a search query using an LLM; and (2) performing a search using the search query, and then the process proceeds to step S5. (1) An input prompt is generated that quotes the content of the user's inquiry in step S1 (generally a response request) and the part of the response text (generally a part of the response text) that was determined to be "necessary" in the fact-verification requirement in step S3. The LLM unit 7 is instructed to generate a search query for the second DB41 to verify the facts about the "necessary" part, and the search query is obtained as an output prompt. (2) The second DB 41 is searched using the above automatically generated search query, and the documents of the search results (documents to be used as sources for fact verification) are output to the second generation unit 5.

[0038] The second DB 41 may be constructed in advance as a database that stores documents containing factual descriptions so that the database can be searched by the query. For example, if the first DB 11 is made up of case documents about consultations on contracts and operation methods for mobile phones, etc., the second DB 41 may be constructed in advance as a database that contains documents describing the specific contents of various contracts for mobile phones, etc., and documents such as manuals describing specific operation methods.

[0039] 10 shows a schematic example EX40 of the input / output prompt and search results in the second search unit 4. The search results in this example EX40 are the same as those in example EX5 of FIG.

[0040] Step S4 (and step S5) when additional fact-verification processing is not performed in step S3 In the above description, the additional processing in step S3 is to determine whether fact verification using the LLM unit 7 is necessary, but if this additional processing is omitted, the same process can be carried out in step S4 by treating all the necessity in the processing of (1) above as "necessary." The same process can be carried out in the next step S5 by treating all the necessity as "necessary."

[0041] Step S5: Second Generation Unit 5 In step S5, the second generation unit 5 issues an instruction to the LLM unit 7 to generate a response statement (generally a response statement) in response to the user's inquiry based on the source document for the strategy linked to each part, as an input prompt citing the inquiry content (generally a response request statement) input by the user in step S1, the response statement (generally a response statement) to which a strategy has been assigned for each part obtained in step S3 and to which a determination has been made as to whether fact verification is necessary, and the document that serves as the source of fact verification for the part obtained in step S4 for which the fact determination is ``necessary,'' thereby generating a final response statement (generally a response statement) to the user as an output prompt, sending it to the user, and then ending the flow of Figure 6.

[0042] In addition, the answer sentence as an output prompt automatically generated by the second generation unit 5 as described above may be manually checked by an operator, and if necessary, may be modified and sent to the user.

[0043] As shown in FIG. 11, a schematic example EX50 of an input / output prompt in the second generation unit 5 may be used as an input prompt, for example, "Please generate an answer sentence based on the information obtained from the customer support manual in accordance with the strategy list." In other words, the input prompt may be configured as an instruction to generate an "answer sentence" to a user inquiry that realizes a "strategy linked to each part (strategy list)" based on a "source document (customer support manual)." Note that the output example in this example EX50 is the same as example EX6 in FIG. 5. Note that in FIGS. 10 and 11, the parts written in white on a black background in reverse video indicate the content of each process as a catchphrase, and do not represent input / output prompts or search results.

[0044] As described above, according to the embodiment of the present invention, when automatically generating a response using response history (past cases in the first DB11), if the response is not anonymized, differences from past cases will affect the response, and if the response is anonymized too much, it is expected that the response will not be effectively utilized.However, it is possible to automatically generate an appropriate response by abstracting the differences from past cases in a way that does not (excessively) anonymize them (abstraction achieved by processing from the strategy decomposition unit 3 onwards).

[0045] Various supplementary and alternative examples will be explained below.

[0046] (1) The embodiments of the present invention can provide greater convenience for RAG, which has been actively researched and developed recently as a powerful application of LLM, a type of generative AI (artificial intelligence) that automatically generates text. This can contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote inclusive and sustainable industrialization, and promote industrialization and inclusive technology."

[0047] (2) Although not shown in Example EX30 of Figure 8, when the strategy decomposition unit 3 instructs the LLM unit 7 to identify and assign strategies for each part, an additional consideration may be to instruct the LLM unit 7 to assign a more abstract and / or general strategy (i.e., a more general strategy) when the part and / or strategy is related to personal information compared to when it is not related to personal information. This may promote the effect of assigning a strategy that broadens its scope of application without limiting it to specific cases related to personal information. That is, as shown in the example of Figure 5 above, the age of the child being consulted about the contract content was specifically stated as "8 years old" in the provisional response generated by the first generation unit 2, whereas the final response generated by the second generation unit 5 broadens its scope of application by categorizing it into "under 18 years old" and "over 18 years old." This broadens the scope of application of the response (response) by transforming the provisional response into the final response, thereby avoiding the provisional response from becoming too specific. In order to (more reliably) obtain the effect of widening the scope of application, the input prompt may be configured as an instruction to assign a strategy in a more general form when a part and / or its strategy is related to personal information in the strategy decomposition unit 3. The input prompt may include, for example, the statement, "For parts and strategies that are considered to be related to personal information, please assign strategies in a more general way."

[0048] (3) FIG. 12 is a diagram showing an example of the hardware configuration of a general computer device 70. The terminal 10 and the information processing device 20 in the information processing system 100 can each be realized as one or more computer devices 70 having such a configuration. When the terminal 10 or the information processing device 20 is realized using two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, one or more dedicated processors 72, such as a GPU (graphics processing unit) and / or an NPU (neural network processing unit), that execute some or all of the CPU 71's execution instructions in place of or in cooperation with the CPU 71 and are specialized for specific operations, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the dedicated processor 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76 that provides display output, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, or the like, a speaker 78 that outputs audio, a microphone 79 that inputs audio, and a bus BS for transmitting and receiving data among these.

[0049] Each functional unit of the information processing device 20 can be realized by a CPU 71 and / or a dedicated processor 72 that reads from a ROM 74 and executes a predetermined program corresponding to the function of each unit. Both the CPU 71 and the dedicated processor 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and dedicated processor 72. When communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and dedicated processor 72. When audio input / output-related processing is performed, a speaker 78 and a microphone 79 also operate in conjunction with the CPU 71 and dedicated processor 72.

[0050] When inputting and outputting text data and voice by a user or operator in the terminal 10 or the information processing device 20, a display 76, an input interface 77, a speaker 78, a microphone 79, etc. may be used. Voice input data may be converted into text using existing voice analysis technology, and text input may be accepted. [Explanation of symbols]

[0051] 100...information processing system, 10...terminal, 20...information processing device, 1...first search unit, 11...first DB, 2...first generation unit, 3...strategy decomposition unit, 4...second search unit, 41...second DB, 5...second generation unit, 7...LLM unit

Claims

1. a first process of searching a first database storing documents of solved cases for documents similar to the request from the user; a second process of generating a first response sentence from the similar document and the response request sentence using a large-scale language model; a third process of assigning a strategy to each part of the first response sentence using the large-scale language model; a fourth process of searching, using the large-scale language model, for a verification source document for verifying facts for each part to which the strategy is assigned from a second database storing documents including statements of fact; a fifth process of generating a second response sentence in response to the response request sentence from the verification source document and the first response sentence to which a strategy has been assigned for each part, using the large-scale language model; In the third process, when assigning the strategy, an instruction is given to the large-scale language model to increase the generality of the strategy related to personal information.

2. In the third process, the large-scale language model is further used to determine whether or not fact verification is required for each of the parts; 2. The information processing apparatus according to claim 1, wherein in the fourth process, the search is performed on a portion determined in the third process to require fact verification.

3. a first step of searching for a similar document containing a description of a similar request from a first database storing documents of solved cases in response to a response request from a user; a second step of generating a first response sentence from the similar document and the response request sentence using a large-scale language model; a third step of assigning a strategy to each part of the first response sentence using the large-scale language model; a fourth step of searching, using the large-scale language model, for a verification source document for verifying facts for each part to which the strategy is assigned from a second database storing documents containing factual statements; a fifth step of generating a second response sentence in response to the response request sentence from the verification source document and the first response sentence to which a strategy has been assigned for each part, using the large-scale language model, The method is characterized in that in the third step, when assigning the strategy, an instruction is given to the large-scale language model to assign a strategy that is more general with respect to things related to personal information.

4. a first process of searching a first database storing documents of solved cases for documents similar to the request from the user; a second process of generating a first response sentence from the similar document and the response request sentence using a large-scale language model; a third process of assigning a strategy to each part of the first response sentence using the large-scale language model; a fourth process of searching, using the large-scale language model, for a verification source document for verifying facts for each part to which the strategy is assigned from a second database storing documents including statements of fact; a fifth process of generating a second response sentence in response to the response request sentence from the verification source document and the first response sentence to which a strategy is assigned for each part, using the large-scale language model, The program is characterized in that, in the third process, when assigning the strategy, an instruction is given to the large-scale language model to increase generality of the strategy related to personal information.

Images