Information processing device, information processing method, and program

Abstract

Improve the accuracy of answers to user questions. [Solution] The information processing device 100 includes a question acquisition unit 21 that acquires a question from the user, a character information acquisition unit 22 that acquires a plurality of information sets composed of character information, a character information search unit 23 that searches for a plurality of information sets that have a high similarity to the question from the plurality of information sets, a rank acquisition unit 24 that inputs the searched plurality of information sets and an instruction sentence including the question from the user to a language model 114 and acquires a rank indicating the degree of similarity between the plurality of information sets and the question, and an answer output unit 25 that outputs an answer to the question based on the information sets and the rank.

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program. [Background Art]

[0002] Patent Document 1 discloses an information processing system in which a processor of an employee terminal identifies past corresponding data via an employee's operation on an input unit, receives input of machine learning conditions, and transmits the machine learning conditions to a bot server, wherein the bot server changes machine learning conditions based on a learning model in accordance with the received machine learning conditions, acquires vector information from at least one of a question sentence and an answer sentence, and requests a request for related information based on at least one content of the question sentence and the answer sentence. Further, the related information server disclosed in Patent Document 1 identifies related information in response to receiving a request for related information from the bot server, and transmits the identified related information to the bot server. The bot server transmits, based on the acquired vector information, improvement measures that are classification results and the related information to the employee terminal. [Prior Art Document] [Patent Document]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2024-100098 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] Conventionally, automatic answering systems that answer online user questions online have been known. In such systems, it is required to improve the accuracy of answers.

[0005] This invention has been made in view of the above-mentioned problems, and aims to provide an information processing device, an information processing method, and a program that can improve the accuracy of answers to user questions. [Means for solving the problem]

[0006] According to one aspect of the present invention, the information processing device includes: a question acquisition unit that acquires a question from a user; a character information acquisition unit that acquires a plurality of information sets composed of character information; a character information search unit that searches for a plurality of information sets that have a high degree of similarity to the question from the plurality of information sets; a rank acquisition unit that inputs the retrieved plurality of information sets and an instruction sentence including the question from the user into a language model and acquires a rank indicating the degree of similarity between the plurality of information sets and the question; and an answer output unit that outputs an answer to the question based on the information sets and the rank.

[0007] According to one aspect of the present invention, an information processing method includes the steps of: obtaining a question from a user; obtaining a plurality of information sets composed of character information; searching for a plurality of information sets from the plurality of information sets that have a high similarity to the question; inputting the searched plurality of information sets and an instruction sentence including the question from the user into a language model to obtain a ranking indicating the degree of similarity between the plurality of information sets and the question; and outputting an answer to the question based on the information sets based on the ranking.

[0008] According to one aspect of the present invention, a program configured to be executable by a computer causes the computer to perform the following steps: to obtain a question from a user; to obtain a plurality of information sets consisting of character information; to search for a plurality of information sets from the plurality of information sets that have a high similarity to the question; to input the searched plurality of information sets and an instruction sentence containing the question from the user into a language model to obtain a ranking indicating the degree of similarity between the plurality of information sets and the question; and to output an answer to the question based on the information sets based on the ranking. [Effects of the Invention]

[0009] According to the present invention, it is possible to answer user questions with higher accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] This is a schematic diagram showing the configuration of an information processing device according to an embodiment of the present invention. [Figure 3] This figure shows an example of a data table stored in the storage unit of an information processing device according to an embodiment of the present invention. [Figure 4] This is an image diagram showing an example of a screen displayed on a user terminal according to an embodiment of the present invention. [Figure 5] This is a flowchart illustrating the operation of an information processing device according to an embodiment of the present invention. [Figure 6] This figure shows an example of a dataset used to evaluate embodiments of the present invention. [Figure 7] This figure shows the evaluation results of embodiments of the present invention. [Modes for carrying out the invention]

[0011] Hereinafter, an information processing apparatus 100, an information processing method, a program, and an information processing system 1000 according to an embodiment of the present invention will be described with reference to the drawings.

[0012] First, the configuration of the information processing system 1000 will be explained with reference to Figures 1 to 4. Figure 1 is a schematic diagram showing the configuration of the information processing system 1000. Figure 2 is a schematic diagram showing the configuration of the information processing device 100. Figure 3 is a diagram showing an example of a data set stored in the storage unit 10 of the information processing device 100. Figure 4 is a diagram showing an example of a screen displayed on the user terminal 120.

[0013] The information processing system 1000 is a system for providing answers to user questions. The information processing system 1000 comprises an information processing device 100, a server device 110, and a user terminal 120.

[0014] The information processing device 100 is comprised of a computer including a CPU 101 (central processing unit), a storage device 102 consisting of ROM (read-on memory) and RAM (random access memory), and an I / O interface 103 (input / output interface). The RAM stores data for processing by the CPU 101, the ROM stores control programs for the CPU 101 in advance, and the I / O interface 103 is used for inputting and outputting information to and from connected devices. The CPU 101 executes various processes of the information processing device 100 as described herein by executing the control program. The information processing device 100 may be configured as a single device, or it may be divided into multiple devices, with each control being distributed among these multiple devices.

[0015] The information processing device 100 is configured to communicate with the server device 110 and the user terminal 120 via a network NW.

[0016] The information processing device 100 functions as a server that provides web pages to users via the internet. The information processing device 100 displays web pages on the user terminal 120 in response to requests from users via the user terminal 120. Furthermore, the information processing device 100 has a function to automatically output answers to user questions entered on the web page.

[0017] As shown in Figure 2, the information processing device 100 comprises a storage unit 10 for storing information and a processing unit 20 for executing various processes. Note that the configurations of the information processing device 100 shown in Figure 2 represent the functions of the information processing device 100 as virtual units (functional units) and do not necessarily mean that they exist physically.

[0018] The storage unit 10 is configured by, for example, a database, and stores in advance information necessary for processing executed by the processing unit 20. Specifically, the storage unit 10 stores a plurality of information sets configured from character information, which indicate information for answering questions from users. That is, an information set means a collection of character information.

[0019] In the present embodiment, the plurality of information sets include information useful for accounting services. Specifically, the plurality of information sets include information on at least any one of accounting, final tax return, blue return, invoice preparation, payroll calculation, side business, foundation or opening of a business, tax consultant consultation, company merger, acquisition or business succession, the Act on Electronic Bookkeeping, the invoice system, and funding. Note that the information sets are not limited to those related to accounting services, and may be information of various attributes including information for corporate management support.

[0020] In the present embodiment, character information published on web pages is used as the information set. Character information published on one or more web pages can be configured as one information set. As shown in FIG. 3, in the storage unit 10, character information of each information set, a unique ID, and addresses of one or more web pages on which the character information of the information set is published are stored in association with each other. For example, an information set can be created from information provision pages (hereinafter also simply referred to as "articles") such as blogs and columns published on a company's website (on a specific domain). As shown in FIG. 3, an information set is composed of an article title and content.

[0021] The information set is not limited to character information published on web pages, and can be arbitrarily set. For example, it may be character information independently created by an operator operating the information processing apparatus 100 to answer user questions, or may be one based on information described in publicly known literatures and books.

[0022] As shown in Figure 2, the processing unit 20 includes a question acquisition unit 21, a character information acquisition unit 22, a character information search unit 23, a ranking acquisition unit 24, an answer output unit 25, and a display control unit 26.

[0023] The question acquisition unit 21 acquires text indicating a question from the user. Specifically, the question acquisition unit 21 extracts text indicating the user's question (hereinafter also referred to as "question text") from the request signal input from the user terminal 120. The question text acquired by the question acquisition unit 21 is input to the character information retrieval unit 23.

[0024] The character information acquisition unit 22 acquires multiple information sets composed of character information. Specifically, when a request signal including a user's question is input from the user terminal 120, the character information acquisition unit 22 acquires multiple information sets from the storage unit 10. The information sets acquired at this time may be all the information sets stored in the storage unit 10, or some or more information sets extracted from all the information sets stored in the storage unit 10 based on predetermined rules, etc. In this embodiment, the character information acquisition unit 22 will be described as acquiring all the information sets stored in the storage unit 10. The multiple information sets acquired by the character information acquisition unit 22 are input to the character information search unit 23.

[0025] The character information retrieval unit 23 searches for multiple information sets (for example, five) that have a high similarity to the user's question from among multiple information sets. Specifically, the character information retrieval unit 23 converts the question text and information sets representing the user's question into embedding vectors, and is configured to allow the nearest neighbor search engine to search for multiple information sets that have a high similarity to the question text. A publicly known nearest neighbor search engine can be used, but for example, Meta's FAISS (Facebook AI Similarity Search) algorithm can be used. Hereinafter, the multiple information sets that have a high similarity to the user's question, retrieved by the character information retrieval unit 23, will also be referred to as the "similar information set group".

[0026] The ranking unit 24 inputs an instruction statement (prompt) containing a group of similar information sets and a question text indicating a question from the user to the language model 114 of the server device 110, which will be described later, and obtains a ranking indicating the degree of similarity between the question and the multiple information sets included in the group of similar information sets. Specifically, the ranking unit 24 sends an instruction statement as a request signal to the server device 110, which is equipped with the language model 114, containing the question text, a group of similar information sets, an instruction to rank the information sets included in the group of similar information sets in order of similarity to the question text, and an output instruction for the reason for determining the ranking. The ranking unit 24 also receives an output signal from the server device 110 containing the ranking result and the reason for the ranking determined by the language model 114 based on the instruction statement. The ranking result of the group of similar information sets obtained by the ranking unit 24 is input to the answer output unit 25.

[0027] The answer output unit 25 outputs an answer to the user's question based on the similar information set group and the ranking obtained by the ranking acquisition unit 24. Specifically, the answer output unit 25 outputs information indicating the address of the web page where the information set with the highest ranking obtained by the ranking acquisition unit 24 from the similar information set group (in other words, the information set deemed to have the highest similarity by the language model 114) is disclosed, along with the answer to the user's question. The answer output by the answer output unit 25 is input to the display control unit 26.

[0028] The display control unit 26 displays a chat screen 126 on the user terminal 120 in response to a user request, which serves as an interface for receiving user questions and providing answers (see Figure 4). The chat screen 126 includes a question input field 127 and a response field 128 where the answer to the question is displayed. The chat screen 126 is displayed on the display device 125 of the user terminal 120, which will be described later, for example, by accessing a web page on the internet. In this way, the display control unit 26 is configured to perform interactive communication with the user terminal 120 via chat.

[0029] More specifically, when the display control unit 26 receives a request from the user terminal 120 to display a chat screen based on user operation, it sends a command signal to the user terminal 120 to display the chat screen 126 (input field 127 and response field 128) and accepts questions from the user through the user terminal 120. When the user enters a question into the input field 127, the display control unit 26 sends a command signal to the user terminal 120 to display the input content in the response field 128. Furthermore, when the display control unit 26 receives an answer to the question from the answer output unit 25, it sends a command signal to the user terminal 120 to display the answer in the response field 128. After sending the command signal to display the answer, the display control unit 26 again accepts questions from the user through the input field 127.

[0030] As shown in Figure 1, the server device 110 is composed of a computer equipped with a CPU 111 (central processing unit), a storage device 112 consisting of ROM (read-on memory) and RAM (random access memory), and an I / O interface 113 (input / output interface). The RAM stores data for processing by the CPU 111, the ROM stores control programs for the CPU 111 in advance, and the I / O interface 113 is used for inputting and outputting information with connected devices. The CPU 111 executes the control programs to perform various processes of the server device 110 as described herein. The server device 110 may be configured as a single device, or it may be divided into multiple devices, with each control being distributed among these multiple devices.

[0031] The server device 110 is a server in which a language model 114 is stored in a storage device 112. Based on an instruction input from an external source, the CPU 111 executes a task using the language model 114 retrieved from the storage device 112 and outputs the execution result. The language model 114 consists of a trained model that has learned instruction sentences and their corresponding responses through machine learning. As the language model 114, for example, a large language model (LLM) constructed by training a large amount of text data using deep learning as a machine learning method can be used. Examples of large language models include OpenAI's GPT series, Google's Gemini, and DeepSeek's DeepSeek.

[0032] The user terminal 120 is comprised of a CPU 121 (central processing unit), a storage device 122 consisting of ROM (read-only memory) and RAM (random access memory), and a computer equipped with an I / O interface 123 (input / output interface), an input device 124, and a display device 125. The RAM stores data for processing by the CPU 121, the ROM stores control programs for the CPU 121 in advance, and the I / O interface 123 is used for inputting and outputting information with connected devices. The CPU 121 executes various processes of the user terminal 120 as described herein by executing the control program. The user terminal 120 may be a mobile terminal such as a smartphone, or a personal computer (PC).

[0033] In the user terminal 120, information is displayed on the display device 125 and communication is performed with other devices such as the information processing device 100, in response to user operation inputs received through the input device 124.

[0034] Next, the operation of the information processing device 100 and the information processing system 1000 will be described. Although each process performed by the information processing device 100 is performed by each functional unit, in the following description, the processes performed by each functional unit included in the processing unit 20 will be described as being performed by the processing unit 20.

[0035] When a user attempts to access a webpage containing a chat screen 126 (see Figure 4) on the user terminal 120, the user terminal 120 sends a signal requesting the display of the webpage to the information processing device 100. Upon receiving the display request signal, the processing unit 20 sends a command signal to the user terminal 120 to display the webpage containing the chat screen 126, causing the chat screen 126 to be displayed on the user terminal 120. This allows the user to input any question as text on the chat screen 126.

[0036] When a user terminal 120 enters a question into the input field 127 of the chat screen 126 and inputs an operation to send the question, the user terminal 120 sends a response request signal containing the content of the question to the information processing device 100. When the processing unit 20 receives the response request signal, it executes the process shown in Figure 5.

[0037] The processing performed by the processing unit 20 will be described below with reference to Figure 5.

[0038] First, in step S10, the processing unit 20 obtains (extracts) text information (question text) that represents the question included in the response request signal.

[0039] In step S11, the processing unit 20 retrieves all information sets and corresponding IDs stored in the storage unit 10.

[0040] In step S12, the processing unit 20 searches for information sets similar to the question text obtained in step S10 from the multiple information sets obtained in step S11. Specifically, it uses a nearest neighbor search library capable of performing embedding vector similarity searches to rank the similarity of each information set to the question text. The similarity is, for example, the cosine similarity between the embedding vectors. In step S12, the top 5 information sets with the highest similarity among the information sets found (ranked) in this way are obtained as search results (in other words, a group of similar information sets).

[0041] In step S13, the processing unit 20 creates and inputs an instruction statement to the language model 114 of the server device 110. The instruction statement includes the question text, the information sets and IDs included in the similar information set group, an instruction to output the results of ranking the similar information set group in order of similarity to the question text, and an instruction to output the reason for the ranking. In this way, the instruction statement includes content to output the reason for the ranking based on the concept of CoT (Chain of Thought). This improves the accuracy of the ranking by the language model 114. The instruction statement may also include instructions on the output format of the response output by the language model 114.

[0042] The output of the language model 114 may be limited to outputting only the top few similar information sets from the group of similar information sets (for example, outputting only the most similar information set), or it may be outputting the results of ranking all similar information sets contained in the instruction. In the following explanation, we will assume that the language model 114 outputs the results of ranking all similar information sets.

[0043] When an instruction is input from the information processing device 100 to the language model 114 of the server device 110, the server device 110 outputs the results of ranking similar information sets in order of similarity to the question text according to the instruction, along with the reasons for the ranking.

[0044] In step S14, the processing unit 20 obtains the rank of the similar information set group output from the language model 114 of the server device 110.

[0045] In step S15, the processing unit 20 searches the database of the storage unit 10 using the corresponding ID based on the results obtained in step S14, and obtains the address information of the information set that is most similar to the user's question.

[0046] In step S16, the processing unit 20 outputs to the user terminal 120 the response, including information indicating the link to the address obtained in step S15. Specifically, the processing unit 20 outputs a display signal to the user terminal 120 that causes the link, which is the response, to be displayed in the response field 128 on the user terminal 120.

[0047] In this way, the processing unit 20 automatically generates an answer to the user's question, outputs it to the user, and terminates the process.

[0048] The user can obtain an answer to a question by viewing a set of information (web pages) similar to the question via a link displayed on the user terminal 120.

[0049] The information processing system 1000 and information processing device 100 according to this embodiment have been described above, but the series of processes in the information processing device 100 described above may be provided as a program for a computer to execute them.

[0050] For example, the program according to the above embodiment is a program configured to be executable by an information processing device 100 as a computer, and causes the computer to perform the following steps: to obtain a question from a user; to obtain a plurality of information sets consisting of character information; to search for a plurality of information sets from the plurality of information sets that have a high similarity to the question; to input an instruction sentence including the searched plurality of information sets and the question from the user into a language model 114 to obtain a ranking indicating the degree of similarity between the plurality of information sets and the question; and to output an answer to the question based on the information sets based on the ranking.

[0051] Furthermore, the program for executing the series of processes described above is provided by a storage medium readable by the information processing device 100. Alternatively, the program may be provided to the information processing device 100 via a network NW. Also, a portion of the processing performed by the information processing device 100 as described in the above embodiment may be executed on a user terminal 120 or a server device 110.

[0052] In this embodiment, the information processing device 100 uses the nearest neighbor search library included in the character information retrieval unit 23 to search for multiple information sets similar to the user's question from among multiple information sets, and then obtains the ranking of the searched multiple information sets using the language model 114. The language model 114 can infer the background of the content mentioned in the question and make appropriate connections between words included in the question and words included in the information sets. The language model 114 then ranks the similar information sets, reflecting the inference of the background of the question and the connections between words. Therefore, according to this embodiment, the accuracy of outputting an appropriate answer to the user's question can be improved compared to searching for information sets similar to the user's question using only the nearest neighbor search library.

[0053] Furthermore, in this embodiment, the instructions given to the language model 114 do not generate answers to user questions, but rather rank similar information sets in order of similarity to the question, or select an information set from the similar information sets that will answer the user's question. In other words, this embodiment uses the RAG (Retrieval-Augmented Generation) method, which generates output by referring to the information (similar information sets) input to the language model 114. Therefore, the accuracy of the answers to user questions can be improved compared to when the language model 114 directly generates the answers to user questions.

[0054] In particular, in fields such as accounting, which require complex specialized knowledge and constant updates to information due to legal revisions, the "hallucination" problem where the language model 114 generates inaccurate answers is likely to occur. Furthermore, the language model 114 may also generate inaccurate answers due to bias. From another perspective, if the language model 114 were to accurately answer user questions in such specialized fields, it might be necessary to additionally train and fine-tune the language model 114 with all the information set from this embodiment. In this case, significant resources would be required for training.

[0055] In contrast, in this embodiment, the nearest neighbor search library of the character information retrieval unit 23 and the language model 114 are used to output an answer containing the most similar set of information to the user's question. In this embodiment, the language model 114 generates an answer (ranking) based on the set of information contained in the instruction sentence, based on the RAG method, so the problem of inaccurate answers being generated due to hallucinations, etc., does not occur, and the accuracy of the answer to the user's question can be improved. In addition, since it is not necessary to train the language model 114 with a large amount of data, resource consumption can be reduced.

[0056] Furthermore, in this embodiment, the concept of CoT (Center of Technology) is used, and the instruction includes an instruction to output the reason for the ranking. This improves the accuracy of ranking the information set in response to the user's question.

[0057] Next, a modified example of this embodiment will be described.

[0058] In the above embodiment, the server device 110 is equipped with a language model 114, and the information processing device 100 obtains the ranking of similar information groups from the language model 114 by sending an instruction to the server device 110. Alternatively, the language model 114 may be provided in the information processing device 100.

[0059] In the above embodiment, the instruction statement includes an instruction to output the reason for the ranking based on the concept of CoT (CoT prompting). However, the instruction to output the reason for the ranking (CoT prompting) is not mandatory and does not have to be included in the instruction statement. On the other hand, the instruction statement may also include instructions that show the thought process for ranking, in addition to the reason for the ranking. Furthermore, so-called One-Shot prompting or Few-Shot prompting techniques may be employed instead of or in addition to CoT prompting. For example, in addition to the instruction to output the reason for the ranking, the instruction statement may include a hypothetical question, a hypothetical set of similar information, the ranking of the hypothetical set of similar information, and the reason for that ranking as an example. Alternatively, the reason for the ranking may be included in the answer to the user's question.

[0060] In addition to the configuration of the above embodiment, the instruction statement of the language model 114 may include an instruction to calculate and output the similarity of a set of similar information for a question. The answer to the user may also include this similarity. Furthermore, the search results of the character information retrieval unit 23 may be included in the answer to the user's question. For example, the answer to the user's question may include the similarity as a search result of the character information retrieval unit 23 (neighborhood search library), or it may include the ranking results of the set of similar information by the character information retrieval unit 23 and the set of similar information by the language model 114, or it may include the degree of agreement between the ranking results of the character information retrieval unit 23 and the language model 114. With these, the user can evaluate the reliability or validity of the answer provided by the information processing system 1000.

[0061] In the above embodiment, the answer to the user's question includes address information (link) of the web page where the information set is disclosed. In contrast, the answer to the user's question is not limited to a web page address and may be based on a ranked information set. For example, the answer to the user's question may be part or all of the content of the information set (content disclosed on the web page) that is similar to the question, a summary of the content of the information set, or something generated by referring to the content of the information set.

[0062] Furthermore, in the above embodiment, the response to the user is based on the set of information that the language model 114 has determined to be the most similar. In contrast, the response to the user is not limited to a configuration based on a single set of information, as long as it is output based on the ranking output by the language model 114. For example, the response to the user may be provided with multiple sets of similar information, all or part of the ranking output by the language model 114. Alternatively, the user may first be provided with a response based on the set of information that is most similar to the question, the user may respond whether the answer is sufficient, and if the answer is insufficient, a response based on the next most similar set of information may be provided.

[0063] In the above embodiment, the character information retrieval unit 23 of the processing unit 20 searches for a group of similar information sets to the user's question, and the language model 114 ranks the similarity of the group of similar information sets to the user's question. In contrast, the information processing device 100 and the information processing method can also be configured to have the language model 114 generate a hypothetical question based on the user's question, and use the hypothetical question to search for an information set that answers the user's question from multiple information sets. In this modified example, for example, the instruction sentence to the language model 114 does not include the user's question, but includes information on a group of similar information sets and an instruction to generate a hypothetical question based on the user's question for which the group of similar information sets is the answer. The information processing device 100 obtains the hypothetical question from the language model 114, and the character information retrieval unit 23 searches again for an information set similar to the hypothetical question from multiple information sets stored in the storage unit 10. The information processing device 100 then outputs an answer to the user's question that includes the information set most similar to the hypothetical question found by the character information retrieval unit 23. In this modified version, even if the set of similar information outputs to the language model 114 does not contain the most appropriate set of information as an answer to the user's question, the character information retrieval unit 23 can still search for the most appropriate set of information during a similarity search (re-search) of the virtual question, thereby improving the accuracy of the answer to the user's question.

[0064] Next, the evaluation of this embodiment will be described with reference to Figures 6 and 7. Figure 6 is a diagram showing an example of the dataset used for evaluation. Figure 7 is a diagram showing the evaluation results of this embodiment.

[0065] [conditions] The evaluation of this embodiment was carried out under the following conditions.

[0066] (1) Evaluation data (information set and evaluation questions) The information set is generated based on explanatory pages of accounting information published on the web. Specifically, as shown in Figure 6, the text information contained in the following article on the Yayoi Corporation website (https: / / www.yayoi-kk.co.jp) was used to construct the information set. Note that the article is current as of January 6, 2025. • 295 articles on "Useful Accounting Information" (https: / / www.yayoi-kk.co.jp / kaikei / oyakudachi / ). • 220 articles on "Find, learn about, and rely on experts for fundraising methods" (https: / / shikin.yayoi-kk.co.jp / study / index.html). • 723 articles on "Yayoi News" (https: / / media.yayoi-kk.co.jp / ).

[0067] Furthermore, anticipating questions that users might ask, we generated the following evaluation questions.

[0068] First, we extracted articles from each of the following sections—"Useful Accounting Information," "Finding, Learning, and Relying on Experts for Funding," and "Reports"—that could be used to generate questions through manual evaluation. Articles deemed unusable were those that fell under one of the following categories: • Articles that combine multiple elements, with a low proportion of the article dedicated to addressing the question. • Articles that mention similar content, making them highly likely to cause confusion during searches. Articles that are intended to direct users to videos or similar content, and therefore contain insufficient information.

[0069] Next, for each extracted article, instructions were given to generate questions that could elicit answers, and these questions were generated using an LLM (Limited Literacy Model). The GPT-4o model from OpenAI was used as the LLM. The instructions provided defined the following requirements: • Outputs 3 possible questions. • Assumes questions from the general public. Avoid using technical jargon as much as possible.

[0070] From the three potential questions resulting from this process, the most appropriate question was selected. Similar to the selection of articles, the selection was based on the evaluators' judgment, and questions with the following characteristics were rejected during the question selection process. • Questions that do not correspond to the article. • Questions that include unreferenced context. • Questions about how to use accounting software, unrelated to the article's topic.

[0071] Articles concerning the use of accounting software were excluded because many articles contained similar content. Furthermore, articles where all three generated questions were inappropriate were treated as unusable, even if the content itself was usable. The above question generation process generated 260 questions for "Useful Accounting Information," 190 questions for "Finding, Learning, and Relying on Experts for Funding," and 594 questions for "Reports," as shown in Figure 6. As a result, combinations of multiple information sets and evaluation questions for which those information sets are answers are generated as evaluation data.

[0072] (2) Libraries and models used The nearest neighbor search library used in the character information retrieval unit 23 was FAISS from Meta Inc. The embedding model used was Ruri (H. Tsukagoshi and R. Sasano. Ruri: Japanese general textembeddings. arXiv e-prints, pp. 1-14, 2024.). The language model 114 used was gemini-1.5-pro from Google Inc.

[0073] [Examples and Comparative Examples] Example 1 is configured according to this embodiment, and the instruction sentence to the language model 114 includes an instruction to output the reason for the ranking (CoT prompting is performed).

[0074] Example 2 is a modified version of this embodiment, in which the instruction sentence to the language model 114 does not include an instruction to output the reason for the ranking (i.e., no CoT prompting is performed). In other words, the difference between Example 1 and Example 2 is whether or not the instruction sentence to the language model 114 includes an instruction to output the reason for the ranking.

[0075] The comparative example differs from Examples 1 and 2 in that it simply searches for information sets similar to the question using the character information retrieval unit 23 (neighborhood search library) and does not perform ranking by the language model 114. In other words, the comparative example is the search result in step S12 in Figure 5.

[0076] In Example 1, Example 2, and the Comparative Example, the number of similar information sets is five, and they are identical to each other.

[0077] [Evaluation Results] The examples and comparative examples were evaluated using Top-k Accuracy (k=1~5). The results are shown in Figure 7.

[0078] As shown in Figure 7, when comparing the examples and comparative examples, Examples 1 and 2 generally show higher accuracy rates than the comparative examples. Furthermore, Example 1, which incorporates the concept of CoT, shows even greater accuracy rates compared to Example 2, and in particular, when k=1, it showed higher accuracy rates than other methods in experiments using any of the evaluation datasets.

[0079] As described above, it was clear that Examples 1 and 2 had higher accuracy (higher correct answer rate) in answering user questions compared to the comparative example. Furthermore, it was clear that Example 1 had higher accuracy (higher correct answer rate) in answering user questions compared to Example 2.

[0080] The effects and advantages of this embodiment will be described below.

[0081] The information processing device 100 includes a question acquisition unit 21 that acquires a question from the user, a character information acquisition unit 22 that acquires multiple information sets composed of character information, a character information search unit 23 that searches for multiple information sets that have a high similarity to the question from the multiple information sets, a rank acquisition unit 24 that inputs the searched multiple information sets and an instruction sentence containing the question from the user into a language model 114 and acquires a rank indicating the degree of similarity between the multiple information sets and the question, and an answer output unit 25 that outputs an answer to the question based on the information sets and the rank.

[0082] The information processing method includes the steps of: obtaining multiple information sets composed of character information; obtaining a question from a user; searching for multiple information sets from the multiple information sets that have a high similarity to the question; inputting the searched multiple information sets and an instruction sentence containing the question from the user into a language model 114 to obtain a ranking indicating the degree of similarity between the multiple information sets and the question; and outputting an answer to the question based on the information sets based on the ranking.

[0083] A program configured to be executable by a computer causes the computer to perform the following steps: to obtain a question from the user; to obtain multiple information sets consisting of character information; to search for multiple information sets from the multiple information sets that have a high similarity to the question; to input the searched multiple information sets and an instruction sentence containing the question from the user into a language model 114 to obtain a ranking indicating the degree of similarity between the multiple information sets and the question; and to output an answer to the question based on the information sets based on the ranking.

[0084] In these configurations, the instructions to the language model 114 are to rank sets of similar information in order of similarity to the user's question. In other words, these configurations use the RAG method, which generates output by referencing the information input to the language model 114. Therefore, the accuracy of the answers to the user's question can be improved compared to when the language model 114 directly generates the answers to the user's question.

[0085] Furthermore, in the information processing device 100, information processing method, and program, multiple information sets include information relating to any of the following: accounting, tax return filing, blue return filing, invoice creation, payroll calculation, side jobs, starting or opening a business, consulting with a tax accountant, corporate mergers, acquisitions or business succession, the Electronic Bookkeeping Law, the invoice system, and fundraising.

[0086] In this configuration, the information set includes information useful for accounting tasks, allowing for accurate answers even to user questions that require specialized accounting knowledge.

[0087] Furthermore, in the information processing device 100, information processing method, and program, the rank acquisition unit 24 inputs an instruction statement that further includes an output instruction for the reason for determining the rank to the language model 114, and acquires the rank and the reason for determining the rank.

[0088] In this configuration, the concept of CoT is used to create the instruction sentences, making it easier to improve the accuracy of the ranking output by the language model 114.

[0089] Furthermore, the information processing device 100 includes a display control unit 26 that displays a chat input field 127 and a response field 128 for the user. The question acquisition unit 21 acquires the text entered in the input field 127 as text indicating a question, and the answer output unit 25 displays the answer to the question based on the information set in the response field 128.

[0090] In this configuration, questions can be entered by users via chat, and answers can be automatically generated. Users can easily and instantly obtain answers to their questions through chat.

[0091] Although embodiments of the present invention have been described above, these embodiments only represent a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. [Explanation of symbols]

[0092] 100 Information Processing Devices 21 Question acquisition part 22 Character information acquisition section 23. Character Information Retrieval Section 24 Rank acquisition part 25. Answer Output Section 26 Display Control Unit 114 Language Models 127 Input fields 128 Response section

Claims

1. A question acquisition unit that acquires questions from users, A character information acquisition unit that acquires multiple information sets composed of character information, A character information retrieval unit that searches for multiple information sets that have a high similarity to the question from among multiple information sets, A ranking unit inputs the multiple information sets that have been retrieved and the instruction sentences containing the questions from the user into a language model and obtains a ranking that indicates the degree of similarity between the multiple information sets and the questions. The system includes an answer output unit that outputs an answer to the question based on the information set and the ranking. Information processing device.

2. The aforementioned sets of information include information relating to any of the following: accounting, tax returns, blue return filing, invoice creation, payroll calculation, side businesses, starting or opening a business, consulting with a tax accountant, corporate mergers, acquisitions or business successions, the Electronic Bookkeeping Law, the invoice system, and fundraising. The information processing apparatus according to claim 1.

3. The rank acquisition unit inputs the instruction statement, which further includes an output instruction for the reason for determining the rank, to the language model to acquire the rank and the reason for determining the rank. The information processing apparatus according to claim 1.

4. The system further includes a display control unit that displays a chat input field and a response field for the user, The question acquisition unit acquires the text entered in the input field as text representing the question, The response output unit displays the answer to the question based on the information set in the response field. The information processing apparatus according to claim 1.

5. Steps to obtain questions from users, Steps include obtaining multiple information sets composed of character information, The steps include searching for multiple sets of information that have a high similarity to the question from among the multiple sets of information, The steps include inputting the retrieved information sets and instructional sentences containing the questions from the user into a language model to obtain a ranking indicating the degree of similarity between the information sets and the questions, The step of outputting an answer to the question based on the information set, based on the ranking, Information processing methods.

6. A program configured to be executable by a computer, Steps to obtain questions from users, Steps include obtaining multiple information sets composed of character information, The steps include searching for multiple sets of information that have a high similarity to the question from among the multiple sets of information, The steps include inputting the retrieved information sets and instructional sentences containing the questions from the user into a language model to obtain a ranking indicating the degree of similarity between the information sets and the questions, The computer is made to perform the steps of: outputting an answer to the question based on the information set based on the ranking; program.

Images