Information processing device, method, and program

Abstract

An information processing device is provided that can effectively reduce the manual effort required to check LLM (Large Scale Language Model) output sentences. [Solution] The system executes a generation process 3 that generates an output sentence from an input sentence using a large-scale language model, a monitoring process 4 that monitors the changes in each word of the output sentence until each word is determined in the large-scale language model when the generation process is performed, and an information addition process 51 that outputs, linked to the output sentence, points where it is determined that there is a large change in the word changes in the word changes.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, method, and program for executing a large-scale language model (LLM). [Background technology]

[0002] The increasing labor costs of operators and employees for customer support and handling internal inquiries at companies is becoming an issue. The introduction of LLM (large-scale language model), a generative AI for text, is also progressing, but at this stage, humans still need to fact-check the proposed answers generated by generative AI, and there are limits to how much labor costs can be reduced. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] GAO, Yunfan, et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023, 2. Summary of the Invention [Problem to be solved by the invention]

[0004] One existing method that can generate answers specialized for specific fields such as customer support using a general-purpose LLM is a technology called RAG (Extended Search Generation). This technology is disclosed in Non-Patent Document 1 and is widely used in various fields.

[0005] LLM is trained using vast amounts of data and is highly versatile, but applying LLM directly to specific fields (such as customer support or internal inquiries) where the knowledge is not sufficiently contained in the vast amount of data from which it is trained may not necessarily produce appropriate results, and false answers may be obtained, a known issue with LLM known as hallucination.However, the RAG method makes it easy to specialize LLM for specific fields.

[0006] In RAG, necessary information is collected from databases specialized in a specific field, such as the web or manuals, and then input into the LLM prompts to generate answers. However, even with RAG, incorrect or unfounded information may be generated, especially for complex tasks.

[0007] In particular, in use cases such as customer support, user inquiries are often vague or complex, so a single response generated by RAG often contains a large amount of information, which is expected to increase the costs of fact-checking and correction.

[0008] In view of the above-mentioned problems with the conventional technology, the present invention aims to provide an information processing device, method, and program that can effectively reduce the manual effort required when checking LLM output statements, not only when RAG is used in particular but also when LLM is used in general. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the present invention provides an information processing device, method, and program, which are characterized by executing a generation process that generates an output sentence from an input sentence using a large-scale language model, a monitoring process that monitors the changes in each word of the output sentence in the large-scale language model when the generation process is performed until each word of the output sentence is determined, and an information addition process that outputs, linked to the output sentence, points where it is determined that there is a large change in the word changes in the word changes. [Effects of the Invention]

[0010] According to the present invention, by automatically detecting potential correction points in an LLM output statement and linking them to the LLM output statement and outputting them, the manual effort required to check the LLM output statement can be effectively reduced. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a functional block diagram of an information processing device according to an embodiment. [Figure 2] 10 is a flowchart illustrating an operation of an information processing device according to an embodiment. [Figure 3] FIG. 10 is a diagram showing a schematic example from an input layer to an output layer in which a monitoring unit monitors the state and converts it into readable text. [Figure 4] FIG. 10 is a diagram showing a schematic example of an LLM output statement with additional information that is displayed and output to an operator in an output unit, together with the original LLM output statement. [Figure 5] FIG. 10 shows a transformer as an example of an existing LLM network that realizes a response generation unit, and is a diagram for explaining the points monitored by the monitoring unit. [Figure 6] FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1 is a functional block diagram of an information processing device according to an embodiment. The information processing device 10 includes an input sentence generation unit 1 including a user response unit 11, a search unit 12, and a sentence generation unit 13, a database (DB) 2, a response generation unit 3, a monitoring unit 4, an information addition unit 5 including a change calculation unit 51, a correction candidate generation unit 52, and a reference source generation unit 53, and an output unit 6.

[0013] 2 is a flowchart of the operation of the information processing device 10 according to one embodiment, and as shown in the figure, steps S1 to S4 are executed in this order. Below, steps S1 to S4 will be explained, along with details of the processing content of each functional block of the information processing device 10 in FIG. 1.

[0014] Step S1: Input sentence generator 1 and DB2 In step S1, the input sentence generation unit 1 performs a standard response to a user requesting a response, records the process as text, searches DB2 for documents necessary for the user response, generates an LLM input sentence from the text of the response record and the retrieved documents, and then proceeds to step S2. At this time, each unit 11 to 13 of the input sentence generation unit 1 performs the following processing.

[0015] The user response unit 11 can be configured as a so-called chatbot using any existing method, and by repeating the process of asking standard questions to a user who desires customer support or other assistance and obtaining answers in the form of user-entered text or selection from a menu, etc., one or more times, it identifies the matters the user desires to be responded to, and saves the history of questions and answers (the dialogue history on the chatbot) in the form of "Question to user (1) → Answer from user (1) → Question to user (2) → Answer from user (2) → ..." up until the identification as a response record in the form of text, and outputs this response record to the search unit 12 and the sentence generation unit 13.

[0016] The user response unit 11 uses, for example, decision tree analysis or similar rule-based techniques, to prepare a series of questions in a tree structure that categorizes the user situation into more detailed cases as it goes from the root node to the leaf node, and by repeating the questions at the next more detailed leaf node that are categorized according to the user's answer to each question, it can function as a chatbot and obtain a response record.

[0017] The search unit 12 refers to the response record obtained from the user response unit 11, identifies from the response record the matters for which the user wishes to be responded to, searches DB2 for the sentences necessary to realize the response, and outputs the retrieved documents to the sentence generation unit 13.

[0018] DB2 is constructed in advance as a database so that reference sentences can be searched in response to a search request from the search unit 12. For example, if the user has subscribed to a telecommunications device and customer support is to be provided, DB2 is constructed in advance as a database by accumulating information such as a collection of successful cases of past customer support responses and operation manuals for the telecommunications device.

[0019] The search unit 12 can search for one or more documents by referring to the response record obtained from the user response unit 11, extracting one or more keywords for search from the response record on a rule basis, and then performing a keyword search on DB2. For example, if the content of the response record is that a user wants to sign a new contract for a certain model α of smartphone, a search can be performed on DB2 using two words, "smartphone α" and "new contract procedure," as search keywords, and the hit documents can be obtained.

[0020] In addition, by using the results of the search performed by the search unit 12 on DB2 to perform the next process in the sentence generation unit 13, processing conforming to the RAG method can be performed in this embodiment.

[0021] The sentence generation unit 13 generates an LLM input sentence based on rules using the interaction record obtained by the user interaction unit 11 and the document searched by the search unit 12, and outputs it to the response generation unit 3. For example, the LLM input sentence can be generated by arranging the search result document and instruction text that indicates that the search result document should be used as reference information for the problem the user wants to solve in the interaction record, after the text of the interaction record, to form an input prompt for the LLM, and can be configured as shown in the following schematic example.

[0022] <Example of an input prompt (LLM input sentence)> ◆ "Users want to know the procedure for signing up for a new contract for Smartphone α, so please guide them through the procedure." ◆ "There are case documents D1 and D2 retrieved from the database as successful examples of guidance on α's contract procedures, and there is manual document D3 retrieved from the database as a manual for α's contract procedures. Please refer to these documents D1, D2, and D3 when providing guidance as necessary."

[0023] Step S2: Response generation unit 3 and monitoring unit 4 In step S2, the LLM input sentence obtained by the sentence generation unit 13 is input to the response generation unit 3 configured as an LLM, an LLM output sentence is generated and output to the information addition unit 5, and then the process proceeds to step S3. Furthermore, in step S2, in parallel with the generation process by the response generation unit 3, the monitoring unit 4 monitors the internal state of the LLM during the processing process from the LLM input sentence to the response generation unit 3 generating the LLM output sentence, and outputs the state of each layer of the LLM during the generation to the information addition unit 5.

[0024] The response generator 3 is configured as an LLM that is trained in advance using an arbitrary learning model and can realize text responses for general purposes, thereby generating LLM output sentences from LLM input sentences as described above. Note that the LLM input sentences and LLM output sentences are also called input prompts and output prompts, respectively.

[0025] The monitoring unit 4 monitors the state from the input layer to the output layer as the state of internal processing from the input prompt to the output prompt obtained by the response generation unit 3, which is an LLM. Specifically, since the LLM can be realized as a network structure using a transformer or the like, the monitoring unit 4 obtains the state of each layer from the input layer to the output layer in the transformer or the like, converts it into readable text, and outputs the state of each layer to the information addition unit 5.

[0026] 3 is a diagram showing a schematic example of the input layer to output layer where the monitoring unit 4 monitors the state and converts it into readable text, with the horizontal axis representing word position k=1,2,3,4,5 and the vertical axis representing layer number i=1,2,...,7,8, with i=1 being the input layer and i=8 being the output layer. The monitoring unit 4 converts the state s_ik (not shown) of word position k and layer number i into words w_ik, and outputs this as the monitored state to the information addition unit 5. In the example of FIG. 3, as a result of this conversion, the input layer i=1 becomes the word string "ABDFH" and the output layer i=8 becomes the word string "ACEFH."

[0027] Here, the conversion from the state s_ik to the word w_ik can be performed by using existing techniques such as the following Non-Patent Document 2. [Non-patent document 2] nostalgebraist, "interpreting GPT: the logit lens", [online], August 31, 2020, AI Alignment Forum, [searched March 28, 2020], Internet <URL:https: / / www.lesswrong.com / posts / AcKRB8wDpdaN6v6ru / interpreting-gpt-the-logit-lens>

[0028] The response generation unit 3 is not limited to a transformer, and can use an LLM based on any model in which word expressions repeatedly change in an intermediate layer; for example, an RNN (recurrent neural network) using a hidden layer as the changing intermediate layer may be used, and the monitoring unit 4 converts the state s_ik of the intermediate layer into words w_ik and monitors it. As shown schematically in Figure 3, the monitoring unit 4 uses a method such as Non-Patent Document 2 to monitor the inference process of each word, i.e., how the words in the sentence that is finally output are inferred, as the transition of each word w_ik for i = 1 to 8 as shown in the following formula H. w1k→w2k→w3k→w4k→w5k→w6k→w7k→w8k …H

[0029] The target of monitoring by the monitoring unit 4 when the response generation unit 3 is implemented using a transformer will be described later with reference to Figure 5. In the example of Figure 3, the word sequence "ACEFH" is obtained as the final output of the output layer at i=8, but the response generation unit 3 does not obtain the word sequence "ABDFH" at the input layer at i=1, for example, as an intermediate output before obtaining this final output. The word sequence "ABDFH" at the input layer at i=1 is the result of monitoring by the monitoring unit 4, converted from the inference process of each word forming the final output sentence into words that can be interpreted by humans, as shown in formula H above.

[0030] Step S3... Information Addition Unit 5 In step S3, the information addition unit 5 adds additional information to the LLM output statement using the LLM output statement obtained from the response generation unit 3 in step S2 and the state of each layer in readable text format obtained from the monitoring unit 4, and outputs the LLM output statement with the added information to the output unit 6, before proceeding to step S4. Specifically, the functional blocks 51 to 53 of the information addition unit 5 perform the following processing to generate various types of additional information and then add the information to the LLM output statement.

[0031] Change calculation unit 51 and correction candidate generation unit 52 The change calculation unit 51 calculates the change in word w_ik at each position k from the state of each layer i obtained from the monitoring unit 4, that is, in the example of Figure 3, the change in the transition process of formula (H) "w_1k → w_2k → w_3k → w_4k → w_5k → w_6k → w_7k → w_8k" when the word at this position transitions through each layer after position k is fixed, and outputs this to the correction candidate generation unit 52. Specifically, the change can be calculated by the following procedure. Note that the example of Figure 3, i.e., the case where the range of i is 1≦i≦8 and the range of k is 1≦k≦5, is used as an example of variables for explanation.

[0032] ◆Step 1: Pass the distribution of words w_ik (w_i1, w_i2, w_i3, w_i4, w_i5) in each layer i through the softmax function to obtain the probability distribution P_i=(p_i1, p_i2, p_i3, p_i4, p_i5). Step 2: KL divergence is used to calculate the interlayer change D of distributions P_i and P_i+1. KL Calculate (P_i||P_i+1). This calculation is performed for distributions P_1 and P_2, distributions P_2 and P_3, ..., distributions P_7 and P_8 for each of the seven locations (between layer i and layer i+1) where i = 1, 2, ..., 7 in all eight layers. ◆ Step 3: Using the judgment threshold TH, determine whether the interlayer change is "D KL In the inter-layer pair i, i+1 that is determined to be large (P_i||P_i+1)≧TH), the words w_ik, w_i+1 k whose probability value p_ik was the largest in the i-th layer but whose probability value p_i+1 k is no longer the largest in the i+1-th layer are marked as correction candidates and output to the correction candidate generation unit 52.

[0033] The following procedures 3a and 3b are also possible as variations of procedure 3, and correction candidates may be obtained by all or some of procedures 3, 3a, and 3b. For example, if all of procedures 3, 3a, and 3b are used, all of the items marked as correction candidates in any of procedures 3, 3a, and 3b may be obtained as correction candidates.

[0034] ◆Step 3a Using the decision threshold TH, for inter-layer pair i, i+1 where the inter-layer change is determined to be large, i.e., "DKL(P_i||P_i+1)≧TH", the words w_ik, w_i+1 k whose probability value p_ik was not the largest in the i-th layer but whose probability value p_i+1 k became the largest in the i+1-th layer are selected as correction candidates.

[0035] ◆ Step 3b At each position k where a word is placed, the point i=i where the interlayer change is maximum as argmax|p_ik - p_i+1 k| max Words w_ i max k,wi max +1 k is set as the correction candidate. The correction candidate is not limited to the single location where the inter-layer change is the largest, but may be a word at a location where the inter-layer change is determined to be on the upper side of the threshold (for example, the largest location and the second location).

[0036] The correction candidate generation unit 52 generates the word w_ik (or w_i+1 k, etc.) marked in the above step 3, etc., as a correction candidate for the k-th word. In the example of FIG. 3, for example, the word at position k=2 changes from B to C and becomes word C in the output layer, while word B is generated as a correction candidate word, and the word at position k=3 changes from D to E and becomes word E in the output layer, while word D is generated as a correction candidate word. As in this example, after the words w_ik and w_i+1 k are set as correction candidates in step 3, etc., the correction candidate generation unit 52 sets the words from these candidates that are different from the words in the output layer as final correction candidates. (This is because words in the output layer do not function as correction candidates.)

[0037] In addition, when the change calculation unit 51 uses the above procedure 3, the kth word marked in procedure 3 exists as word w_8k in the output layer i=8 in the candidate generation unit 52, but information that this word w_8k may be a word that should be corrected can be obtained as additional information and output to the output unit 6.

[0038] Similarly, the correction candidate generation unit 52 can obtain additional information that the candidate word to be used to specifically correct and replace the correction candidate word w_8k is the original word w_ik that had the largest probability value p_ik when marked in the above procedure 3, and output this information to the output unit 6. Similarly, when procedures 3a and 3b are used, additional information can be obtained as a candidate word to be used to correct and replace something marked with a word different from the output layer.

[0039] The significance of the change calculation unit 51 and the correction candidate generation unit 52, which perform the above-described processing, is as follows. That is, due to the nature of learning, LLM tends to converge to probabilities that are close to known word sequences, and looking only at the state of the final layer may not result in a valid candidate. In other words, as confidence increases toward the final layer, "preconceptions" become stronger, so looking only at the layer near the final layer makes it difficult to obtain a valid correction word, and the result is likely to remain incorrect. To address this issue, the change calculation unit 51 and the correction candidate generation unit 52 focus on the "changes" in the word probability distribution from the input layer to the final layer, rather than just the state of the final layer, by performing steps 1 to 3 above, thereby capturing the "LLM's indecision" and alleviating this problem, thereby obtaining more effective correction candidates.

[0040] In other words, the significance of the change calculation unit 51 and correction candidate generation unit 52 of this embodiment is that when the LLM is in a state of ``confidently making a mistake'' (hallucination state) in the final layer, the correct candidate is likely to be a word that was ``discarded after being unsure and deciding that it was not the correct answer'' in the intermediate layers, and it is possible to obtain estimated results for areas that may be subject to hallucination, which is a general problem with LLM, and correction candidates for the hallucination.

[0041] That is, the inter-story change D calculated by the change calculation unit 51 in step 2 KL (P_i||P_i+1) can be considered to represent the risk of hallucination occurring in the final layer (the greater the inter-layer change, the higher the risk).

[0042] ●Reference source generation unit 53 The referencing unit 53 outputs the parts of the LLM output sentence generated by the response generation unit 3 that originate from the document (called the "referencing document") retrieved from DB2 by the search unit 12, along with the information identified as additional information, and outputs the results to the output unit 6. When the response generation unit 3 is realized by a transformer with an attention mechanism, information on the relationship between each word in the input sentence and each word in the output sentence can be calculated from the embedded representations of the input sentence and the output sentence. For example, a word w in the output sentence outFor the embedding representation of vector v(w out ) and similarly calculate each word w in For the embedding representation of vector v(w in ) and calculate the distance between these vectors | v(w out )- v(w in )| is small by threshold judgment, the word w of the input sentence in is the word w in the output sentence out Therefore, additional information can be output by, for example, the following procedure.

[0043] ◆When proceeding with word prediction, the score of the attention mechanism, which calculates which parts of the input sequence should be focused on up to that point, is used as a reference to align references containing tokens with high scores in the threshold judgment. ◆Word sequences that do not score high in the threshold judgment against the references are treated as general speech without references and are not aligned. ◆If the attention mechanism scores high for multiple references, it is possible to take their ratio into consideration and present the reference ratio, for example, as shown below. Document A (80%), Document B (20%)

[0044] ● Step S4... Output section 6 In step S4, the output unit 6 adds the information added by the information addition unit 5 to the LLM output statement obtained from the response generation unit 3 in steps S2 and S3 above, and outputs the result to the operator. If the operator determines that the LLM output statement with the added information needs to be corrected, the operator makes the correction, and the output unit 6 outputs the corrected LLM output statement to the user, and the flow in Figure 2 ends.

[0045] The output unit 6 is configured as a display in terms of hardware, and is therefore able to visually output the LLM output statement with added information to the operator. When accepting corrections from the operator, the hardware can accept the corrections via an input interface such as a keyboard or mouse. Similarly, when outputting the corrected LLM output statement to the user, the hardware can visually output it using a display.

[0046] Figure 4 shows a schematic example of an LLM output statement with additional information, along with the original LLM output statement, which is displayed and output to the operator in the output unit 6. In Figure 4, an example of the original LLM output statement is shown on the upper side, and an example of the LLM output statement with additional information is shown on the lower side.

[0047] As described above, according to this embodiment, by automatically realizing processes such as indicating the parts of the LLM output statement that the operator should check as additional information, suggesting correction candidates when corrections are necessary, and presenting information on the referenced document, the operator's work of checking for the presence or absence of hallucinations in the LLM output statement can be significantly reduced.Then, by finally outputting the result to the user after the operator has made corrections as necessary, the user can obtain information that is not contaminated by hallucinations, and can receive appropriate customer support, etc.

[0048] Various supplementary and alternative examples of the embodiments of the present invention will be described below.

[0049] (1) The embodiments of the present invention can provide greater convenience for RAG, which has been actively researched and developed recently as a powerful application of LLM, a type of generative AI (artificial intelligence) that automatically generates text. This can contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote inclusive and sustainable industrialization, and promote industrialization and inclusive technology."

[0050] (2) Although the present embodiment has been described with reference to the case where the RAG method is used, the partial configuration of the present embodiment can also be applied to a general LLM that does not use the RAG method. In other words, when the RAG method is not used, the search unit 12, DB2, and reference source generation unit 53 can be omitted from the configuration of FIG. 1, and the configuration can be the same as that of the above embodiment.

[0051] In this case, since the user desires a response to the history of questions and answers recorded in the user response unit 11, text instructing the generation of a response can be generated as an LLM input sentence in the sentence generation unit 13 on a rule basis and input to the response generation unit 3.

[0052] (3) FIG. 5 is a diagram showing a transformer with an attention mechanism disclosed in Non-Patent Document 3 and the like, as an example of an existing LLM network 100 that realizes the response generation unit 3. [Non-Patent Document 3] Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).

[0053] In network 100, the left-hand component, designated by reference numbers in the 200s, processes input sentences, while the right-hand component, designated by reference numbers in the 300s, generates output sentences through the iterative process shown in the figure. In the left-hand component, an input embedding layer 201, a positional encoding process 202, a multihead attention layer 203, an additive normalization process 204, a feedforward (FF) layer 205, and an additive normalization process 206 process the input sentence in this order, and output an embedded representation of the input sentence to a multihead attention layer 305. In the right-hand component, an output sentence is generated by iteratively processing an output embedding layer 301, a positional encoding process 302, a masked multihead attention layer 303, an additive normalization process 304, a multihead attention layer 305, an additive normalization process 306, a feedforward (FF) layer 307, an additive normalization process 308, a linear layer 309, and a softmax (SM) process 310 in this order. Here, the repeated processing is performed at the locations indicated by reference numerals 301 to 308, and the linear layer 309 and SM processing 310 perform processing when obtaining an output statement at the end of the repeated processing.

[0054] The additive normalization processes 204, 206, 304, 306, and 308 skip the inputs from the previous stages at the locations shown in the figure (branching the inputs to reference numbers 203, 205, 303, 305, and 307, respectively) and accept them as inputs to be added, thereby fulfilling the role of a residual net and performing normalization. The output obtained from additive normalization process 308 is input again to output embedding layer 301, and the right-side configuration performs repeated processing, with the output successively growing longer. At the end of the repeated processing, the output from additive normalization process 308 is output to linear layer 309, and the final output sentence can be obtained from SM process 310.

[0055] As indicated by the balloon indicating the monitoring location in the figure, the monitoring unit 4 monitors the output state (the state input to the linear layer 309) from the additive normalization process 308 when the right-side configuration performs iterative processing, thereby obtaining the state s_ik shown in FIG. 3 and converting it into the word w_ik. The monitoring unit 4 mainly monitors the internal state of the iterative processing of the right-side configuration, and can perform the processing shown in FIG. 3. (Note that during the iterative processing, there is no direct sentence of the length of the output sentence, but there is a state corresponding to a sentence of that length, and the final output sentence is generated from that state by the processing of the linear layer 309 and SM process 310 at the end of the iterative processing. Also, even during the iteration, there is no determined word, but as explained in FIG. 3, the monitoring unit 4 can monitor the process of changes in each word (as if there were a determined word) in accordance with the iteration using existing methods such as Non-Patent Document 2.)

[0056] Furthermore, while the example of network 100 in Figure 5 is an example consisting of an encoder configured on the left side and a decoder configured on the right side, the existing LLM network that realizes response generation unit 3 is not limited to the example in Figure 5. It may also be configured to process input and output by being composed only of a decoder, which is currently the mainstream. Various details of the LLM network that constitutes response generation unit 3 can be adopted, but in any case, the process of predicting and determining a word in response generation unit 3 is a series of processes that pass through N layers, including FF layers as sublayers, and project the final vector representation obtained into lexical space. Repeated processes are performed in the sense that the next word after the word thus determined is predicted using the same process (passing through layers 1 to N and projecting the final state into lexical space). The process continues until a special token, such as an "EOS token," which indicates the end of a sentence, is predicted.

[0057] The monitoring unit 4, which uses an existing method such as the aforementioned Non-Patent Document 2 (LogitLens), is a technology that can linguistically monitor and interpret the state of each layer other than the Nth layer stacked inside the model in the response generation unit 3, which uses an LLM network that can take on various configurations as described above (although the words are not actually determined as words, as if there were determined words in the process), and as in the example of Figure 3 mentioned above, it is possible to visualize which words are being referred to for each layer from i=1 to i=7 on the way to the determined output sentence of N=8.

[0058] (3) FIG. 6 is a diagram showing an example of the hardware configuration of a general computer device 70. The information processing device 10 can be realized as one or more computer devices 70 having such a configuration. When the information processing device 10 is realized using two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, one or more dedicated processors 72 such as a GPU (Graphics Processing Unit) and / or an NPU (Neural Network Processing Unit) that execute some or all of the CPU 71's execution instructions in place of or in cooperation with the CPU 71 and are specialized for specific operations, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the dedicated processor 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76 that outputs visual data, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a speaker 78 that outputs audio, a microphone 79 that inputs audio, and a bus BS for transmitting and receiving data among them.

[0059] Each functional unit of the information processing device 10 can be realized by a CPU 71 and / or a dedicated processor 72 that reads from a ROM 74 and executes a predetermined program corresponding to the function of each unit. Both the CPU 71 and the dedicated processor 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and dedicated processor 72. When communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and dedicated processor 72. When audio input / output-related processing is performed, a speaker 78 and a microphone 79 also operate in conjunction with the CPU 71 and dedicated processor 72. [Explanation of symbols]

[0060] 10...information processing device, 1...input sentence generation unit, 11...user response unit, 12...search unit, 13...sentence generation unit, 2...DB, 3...response generation unit, 4...monitoring unit, 5...information addition unit, 51...change calculation unit, 52...correction candidate generation unit, 53...reference source generation unit, 6...output unit

Claims

1. A generation process for generating an output sentence from an input sentence using a large-scale language model; a monitoring process for monitoring the transition of each word in the output sentence until each word is determined in the large-scale language model during the generation process; and an information addition process that outputs, linked to the output sentence, portions determined to have large word changes in the word transition as correction candidate portions in the output sentence.

2. The information processing device described in claim 1, characterized in that the information addition process further outputs, linked to the output sentence, words before or after the change at points where it is determined that the word change in the word transition is large, as correction candidate words for the correction candidate points in the output sentence.

3. further executing an input sentence generation process for generating the input sentence; In the input sentence generation process, the input sentence is generated by including a process of a standard response to be carried out with a user requesting a response and a document searched as a reference to realize the response to the user; 2. The information processing apparatus according to claim 1, wherein the information addition process further adds information to the output sentence that indicates a portion derived from the retrieved document.

4. a generation procedure for generating an output sentence from an input sentence using a large-scale language model; a monitoring procedure for monitoring the transition of each word in the output sentence until each word is determined in the large-scale language model during the generation process; An information processing method characterized by the computer executing an information addition procedure in which points determined to have large word changes in the word transition are linked to the output sentence as correction candidate points in the output sentence and output.

5. A generation process for generating an output sentence from an input sentence using a large-scale language model; a monitoring process for monitoring the transition of each word in the output sentence until each word is determined in the large-scale language model during the generation process; An information processing program characterized by causing a computer to function as an information processing device that executes an information addition process that links and outputs, to the output sentence, points in the output sentence where it is determined that there is a large change in the word in the transition of the word.

Images