Response system, response method, and program

Abstract

To provide a recommendation system etc. that suitably suggests LLMs that can appropriately respond to user requests. [Solution] In a recommendation system, a task execution unit receives a request including text data from a user and generates a first response to the request in a first language model, which is one language model specified by the user from among multiple language models. A rating information acquisition unit acquires rating information for each of the multiple language models from a predetermined database. A recommendation information generation unit selects a recommended language model for generating a response to the request from the multiple language models based on the request, the first response, and the rating information, in a second language model having an execution environment different from that of the first language model. Furthermore, the recommendation information generation unit generates recommendation information in the second language model that includes a message presenting the recommended language model to the user.

Description

Technical Field

[0001] The present disclosure relates to an answering system, an answering method, and a program.

Background Art

[0002] Generative AI (Artificial Intelligence) services using large language models (LLMs (Large Language Models)) that generate answers in response to various requests from users are becoming widespread. Users use any of a plurality of generative AI services. Also, in generative AI services, users select any one of a plurality of LLMs with different characteristics and send requests. In this regard, a plurality of proposals for selectively using a plurality of learned models have been disclosed.

[0003] For example, the technique described in Patent Document 1 stores, as digital clones, learning models that learn the life logs of individual experts and predict the answers of each individual expert, inputs question data into the corresponding digital clones, outputs the predicted answers, and evaluates a plurality of digital clones.

[0004] Also, the technique described in Patent Document 2 selects one of a plurality of learning models according to the content of the obtained question item and the requirements of the questioner, and inputs the question data into the selected learning model.

[0005] The system described in Patent Document 3 includes a speech circuit that automatically speaks according to the speech content of an operator, stores and selects different statistical language models according to a plurality of scenarios, and a speech recognition circuit that recognizes the content after speaking.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

[0007] However, the aforementioned techniques are only applicable to specific learning models. On the other hand, there is a need for the development of techniques to select an LLM that can appropriately respond to user requests.

[0008] In view of the above issues, this disclosure aims to provide a recommendation system, etc., that suitably proposes LLMs that can appropriately respond to user requests. [Means for solving the problem]

[0009] The recommendation system relating to this disclosure comprises a task execution unit, an evaluation information acquisition unit, and a recommendation information generation unit. The task execution unit receives a request from a user, including text data, and causes a first language model, which is one language model specified by the user from among multiple language models, to generate a first response to the request. The evaluation information acquisition unit acquires evaluation information for each of the multiple language models from a predetermined database. The recommendation information generation unit causes a second language model, which has a different execution environment from the first language model, to select a recommended language model from among the multiple language models to generate a response to the request based on the request, the first response, and the evaluation information. Furthermore, the recommendation information generation unit causes the second language model to generate recommendation information, which includes a message presenting the recommended language model to the user.

[0010] The recommendation method described herein involves a computer performing the following processes: The computer receives a request from a user, which includes text data, and causes a first language model, which is one of several language models specified by the user, to generate a first response to the request. The computer obtains evaluation information for each of the several language models from a predetermined database. Based on the request, the first response, and the evaluation information, the computer causes a second language model, which has a different execution environment than the first language model, to select a recommended language model for generating a response to the request from among the several language models. The computer causes the second language model to generate recommendation information, which includes a message presenting the recommended language model to the user.

[0011] The program relating to this disclosure causes a computer to execute the following recommendation method. The computer receives a request from a user, which includes text data, and causes a first response to the request to be generated by a first language model, which is one language model specified by the user from among multiple language models. The computer obtains evaluation information for each of the multiple language models from a predetermined database. Based on the request, the first response, and the evaluation information, the computer causes a second language model, which has a different execution environment from the first language model, to select a recommended language model for generating a response to the request from among the multiple language models. The computer causes the second language model to generate recommendation information, which includes a message presenting the recommended language model to the user. [Effects of the Invention]

[0012] According to this disclosure, a recommendation system, recommendation method, and program can be provided that suitably propose LLMs that can appropriately respond to user requests. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram of the answer system according to the first embodiment. [Figure 2]It is a block diagram of a recommendation system according to the first embodiment. [Figure 3] It is a block diagram illustrating the hardware configuration of a computer. [Figure 4] It is a flowchart of a recommendation method according to the first embodiment. [Figure 5] It is the first diagram showing evaluation information. [Figure 6] It is a diagram showing the flow of information in the recommendation system according to the first embodiment. [Figure 7] It is a diagram showing variations of messages generated by a recommendation information generation unit. [Figure 8] It is a block diagram of a recommendation system according to the second embodiment. [Figure 9] It is a flowchart of a recommendation method according to the second embodiment. [Figure 10] It is a diagram showing the flow of information in the recommendation system according to the second embodiment. [Figure 11] It is a block diagram of a recommendation system according to the third embodiment. [Figure 12] It is a flowchart of a recommendation method according to the third embodiment. [Figure 13] It is the second diagram showing evaluation information. [Figure 14] It is a diagram showing the flow of information in the recommendation system according to the third embodiment. <​​​​​​​​​​​​​​The present invention will be described below through embodiments of the invention, but the invention claimed is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means of solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0015] <Embodiment 1> (Answer System 10) The response system 10 will be described with reference to Figure 1. Figure 1 is a block diagram of the response system 10 according to the first embodiment. The response system 10 receives a request from the user, selects a recommended language model from a plurality of language models to generate a response to the received request, and presents recommendation information regarding the selected recommended language model to the user. The response system 10 is connected to the user terminal 400 via network N1 so as to be able to communicate. The response system 10 mainly consists of a recommendation system 100, a database 200, and a server 300.

[0016] The recommendation system 100 is a computer or server with communication capabilities. The recommendation system 100 is connected to the user terminal 400 via network N1. The recommendation system 100 is also connected to the database 200 and server 300 via network N1. In this way, the recommendation system 100 works in cooperation with the database 200 and server 300 to select the recommended language model described above.

[0017] The recommendation system 100 also accepts, in addition to the request, the specification of a language model for generating the response from the user terminal 400. In this case, the recommendation system 100 causes the specified language model to generate a response to the request. Further details of the recommendation system 100 will be described later.

[0018] Database 200 is a computer, server, or storage device that includes at least non-volatile memory. Database 200 is communicably connected to recommendation system 100 via network N1. Database 200 stores at least evaluation information G200. Evaluation information G200 is information that evaluates each of several language models available to recommendation system 100. Evaluation information G200 is used when recommendation system 100 selects a language model to recommend. Database 200 supplies at least a portion of evaluation information G200 to recommendation system 100 upon request.

[0019] Server 300 is a server that is connected to the recommendation system 100 in a communicative manner. Server 300 includes multiple different language models. Specifically, for example, Server 300 has a first language model 311, a second language model 312, a third language model 313, a fourth language model 314, and so on. These language models are LLMs and are configured to be usable by the recommendation system 100. That is, the language models of Server 300 accept access from the recommendation system 100, generate responses to arbitrary requests, and supply the generated responses to the recommendation system 100.

[0020] The language model possessed by server 300 may have all the functions of a generative AI on server 300. Alternatively, the language model possessed by server 300 may have an API (Application Programming Interface) for enabling a predetermined LLM to function on server 300. In this case, the language model may be configured to perform its generative AI functions in conjunction with an external device.

[0021] The multiple different language models possessed by server 300 are, for example, language models provided by multiple different generative AI service providers. Furthermore, these multiple different language models may be language models provided by the same generative AI service, but different versions of each other. These language models may also possess a function called RAG (Retrieval-Augmented Generation). RAG is a mechanism that searches for external information in text generation by the LLM and uses the search results to generate the answer. RAG is also referred to as "search-augmented generation" or "retrieval-augmented generation." By having RAG, the LLM can generate highly relevant information in response to requests.

[0022] The user terminal 400 is a computer, smartphone, or tablet used by the user. The user terminal 400 has an application installed for accessing the response system 10. The user using the user terminal 400 sends requests to the response system 10 through this application. The user using the user terminal 400 also receives responses to those requests from the response system 10 through this application. Although Figure 1 shows only one user terminal 400, the response system 10 can communicate with multiple user terminals 400. The application for accessing the response system 10 is, for example, a web browser.

[0023] (Recommendation System 100) Figure 2 is a block diagram of the recommendation system according to the first embodiment. The recommendation system 100 mainly consists of a task execution unit 110, an evaluation information acquisition unit 120, and a recommendation information generation unit 130.

[0024] The task execution unit 110 receives a request from the user that includes text data, and causes the first language model 311, which is one language model specified by the user from among multiple language models, to generate a first response to the request. That is, the task execution unit 110 works in cooperation with the server 300 to input the request received from the user to the first language model 311 specified by the user, causing the first language model 311 to generate a first response. In this case, the request received from the user may be called a prompt. In this disclosure, "prompt" is an instruction sentence to be input to the language model to obtain a response. The task execution unit 110 may present the response to the request received from the user to the user terminal 400.

[0025] The evaluation information acquisition unit 120 acquires evaluation information for each of the multiple language models from a predetermined database. In this case, the predetermined database is database 200. The evaluation information G200 stored in database 200 includes evaluations for the first language model 311. The evaluations for the first language model 311 include performance prompts, which are information that includes at least some of the prompts previously entered into the first language model 311, and evaluation information for performance responses, which are responses generated by the first language model 311 to these performance prompts. This allows the recommendation system 100 to refer to how the performance responses to performance prompts received by the first language model 311 are being evaluated. The recommendation system 100 can also refer to similar evaluation information for language models other than the first language model 311.

[0026] The recommendation information generation unit 130 selects a recommended language model from multiple language models to generate a response to the request based on the request, the first response, and the evaluation information G200. The recommendation information generation unit 130 also causes the second language model 312, which has a different execution environment than the first language model 311, to generate recommendation information, including a message presenting the recommended language model to the user.

[0027] More specifically, the recommendation information generation unit 130 inputs, for example, the request received from the user and the first response generated by the first language model 311 into the second language model 312. The second language model 312 is configured to select a recommended language model by referring to the input request and first response, as well as the evaluation information G200, and to generate a message presenting the selected recommended language model to the user.

[0028] In this disclosure, "language models in different execution environments" includes cases where the type of language model itself is different. It also includes cases where the type of language model is the same but the version is different. Furthermore, it includes cases where the type and version of the language model are the same but the conditions instructed when generating a response are different. The conditions instructed when generating a response are, for example, the content of the prompt instructions. In this case, the content of the prompt instructions may include, for example, the policy of the instructions, such as "Please summarize," "Please suggest an idea," or "Please evaluate the text." Such a policy of instructions may also be called a "request type."

[0029] As described above, the recommendation information generation unit 130 causes the second language model 312 to generate recommendation information. If the recommended language model included in the recommendation information is a different language model from the first language model 311, the recommendation system 100 presents the user with a recommended language model that has the potential to generate a more appropriate response to the user's request than the first language model 311. In this case, the user who receives the recommendation information can specify the recommended language model and send a request to the recommendation system 100. This allows the recommendation system 100 to provide the user with an appropriate response.

[0030] Furthermore, if the recommended language model included in the recommendation information is the first language model 311, the recommendation system 100 may present the user with a message indicating that the first language model 311 may generate an appropriate response to the user's request.

[0031] In the recommendation system 100 described above, the language model that the user can specify and the recommended language model that the recommendation information generation unit 130 can select may be set to be different. This is the case, for example, when there are additional language models that the user can use for a fee or free of charge, depending on the type of service the user receives. In this case, for example, the user can use the recommended language model as an optional service. That is, the task execution unit 110 accepts the specification of the first language model 311 from among the multiple language models that the user can specify, and the recommendation information generation unit 130 selects a recommended language model from among multiple language models that exceed the range that the user can specify.

[0032] With this configuration, the recommendation system 100 can effectively appeal to users with optional recommended language models. Furthermore, users of the recommendation system 100 can select additional recommended language models as needed in their actual usage scenarios.

[0033] The recommended language model may be offered as a paid option. In this case, the recommendation system 100 may prompt the user to choose whether or not to use the recommended language model as a paid optional service.

[0034] In generative AI services using LLM, there is a demand from users to receive more appropriate answers to their desired requests. By using the appropriate LLM according to the request, users are more likely to receive answers of higher generation quality. Here, generation quality may include elements such as accuracy, relevance, consistency, and customizability of the answer. Accuracy is an element indicating whether the answer is based on facts. Relevance is an element indicating whether the answer is directly related to the user's question. Consistency is an element indicating whether the answer is logical and consistent. Customizability is an element indicating whether the answer is tailored to the user's individual needs and background.

[0035] For example, suppose a user sends a request asking "What's the weather like in Tokyo today?" and the first language model 311 specified by the user does not have RAG functionality. In this case, the first language model 311 will generate a response such as "It is not possible to find out the weather in Tokyo today." On the other hand, suppose the recommended language model has RAG functionality. In this case, the recommended language model will search for the weather in Tokyo, for example, by using an internet search, and generate a response. The recommendation information generation unit 130 will present the user with the recommended language model that has RAG functionality. As a result, the recommendation system 100 can present the user with a high-quality AI generation service. Furthermore, if the provider of the response system 10 offers a high-quality language model as an option to the user, it is expected that upselling opportunities will be expanded.

[0036] (Example hardware configuration) Figure 3 is a block diagram illustrating the hardware configuration of a computer. The recommendation system 100 described above may have the configuration shown in Figure 3. The computer 1000 has a bus 1010, a processor 1020, memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0037] Bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060 to send and receive data to and from each other. However, the method of connecting the processor 1020 and the other components to each other is not limited to bus connection.

[0038] Processor 1020 is a circuit that includes arithmetic units such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0039] Memory 1030 is a main memory device implemented using RAM (Random Access Memory), etc.

[0040] The storage device 1040 is an auxiliary storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, or ROM (Read Only Memory). The storage device 1040 stores a program for realizing the functions of this disclosure.

[0041] The processor 1020 reads this program into memory 1030 and executes it. This causes the processor 1020 to perform the function corresponding to this program. In other words, the program stored in memory 1030 causes the computer 1000 to perform the function of this disclosure.

[0042] The input / output interface 1050 connects the computer 1000 to a predetermined input / output device. The input / output device is, for example, an input device such as a keyboard, an output device such as a display, or an input / output device in which a touch panel is superimposed on a display.

[0043] The network interface 1060 is an interface for connecting the computer 1000 to a predetermined communication network.

[0044] (Recommendation method) Next, the processes executed by the recommendation system 100 will be described with reference to Figure 4. Figure 4 is a sequence diagram showing the recommendation method according to the first embodiment. The sequence diagram in Figure 4 includes the exchange of information between the user terminal 400 and the recommendation system 100, and the recommendation method executed by the recommendation system 100. In the recommendation method according to this disclosure, the recommendation system 100 executes the processes from step S10 to step S14 below.

[0045] In step S10, the recommendation system 100 receives a language model specification from the user terminal 400.

[0046] In step S11, the task execution unit 110 receives a request containing text data from the user terminal 400. In the example shown in Figure 4, the user terminal 400 sends a message to the recommendation system 100 as request G11, which reads, "Please summarize the following text D. Text D:..."

[0047] In step S12, the task execution unit 110 causes the first language model 311, which is one language model specified by the user from among multiple language models, to generate a first response to the received request. In this case, the task execution unit 110 works in cooperation with the server 300 to cause the first language model 311 to generate the response. That is, the task execution unit 110 supplies the request G11 received from the user terminal 400 to the specified first language model 311. The task execution unit 110 receives the message "Summarizing document D:..." as the first response G21 from the first language model 311. Upon receiving the first response G21, the task execution unit 110 sends the received first response G21 to the user terminal 400.

[0048] In step S13, the evaluation information acquisition unit 120 acquires evaluation information G200 for each of the multiple language models from a predetermined database, i.e., database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit 130.

[0049] In step S14, the recommendation information generation unit 130, in cooperation with the server 300, causes the second language model 312, which has a different execution environment from the first language model 311, to generate recommendation information G22. In this case, the second language model 312 selects a recommended language model from multiple language models to generate a response to the request based on the request G11, the first response G21, and the evaluation information G200. Furthermore, the second language model 312 generates recommendation information G22 that includes a message presenting the recommended language model to the user.

[0050] The processes performed by the recommendation system 100 have been described above. In the processes described above, the recommendation system 100 may have a recommended language model that matches the first language model 311. If the recommended language model matches the first language model 311, the recommendation information generation unit 130 may suppress sending the generated recommendation information to the user terminal 400. By doing so, the recommendation system 100 can prevent the presentation of redundant information to the user. The language model specified by the user in step S10 may be set in advance.

[0051] (Evaluation information G200) Next, we will describe the evaluation information G200 with reference to Figure 5. Figure 5 is the first diagram showing the evaluation information G200. The evaluation information G200 includes the username, performance prompt, performance response, request type, and language model.

[0052] The username is a name used to identify the user operating the user terminal 400. The username may be a predetermined ID (Identifier), or it may be the user's name or company name.

[0053] The performance prompt is input information that includes at least part of the request entered by the user. The performance response is the response generated by the language model in response to the request. The request type is a broad classification of the type of response that the language model should generate based on the performance prompt. The request type can also be described as the content of the instructions given by the performance prompt.

[0054] Request types include, for example, generating summaries, generating code, generating translations, assisting with idea generation, presenting search results, and providing example sentences. Request types can be defined arbitrarily.

[0055] A language model is information that identifies a language model. A language model may be a unique identifier, or it may be the name of the language model, etc.

[0056] Specifically, for example, evaluation information G200 includes the following information: The first piece of information states that when user B sent the request "Please summarize text D" to language model "Model 1-ver.3", the actual response "I will summarize..." was received. The second piece of information states that when user B sent the request "Please summarize text D" to language model "Model 2-ver.2", the actual response "The following is the summarized text..." was received.

[0057] The third piece of information includes the fact that when user C sent a request to the language model "Model 1-ver.3" saying "Please generate code to execute...", the code generated by this language model was obtained as the actual response.

[0058] The fourth piece of information includes the fact that when user C sent a request to the language model "Model 1-ver.2" saying "Please generate code to execute...", the code generated by this language model was obtained as the actual response.

[0059] By including the above-mentioned information, the recommendation information generation unit 130 causes the second language model to generate recommendation information that includes the recommended language model. Note that the evaluation information G200 may include only some of the above-mentioned items. For example, the evaluation information G200 does not need to include the user name. In addition to the above-mentioned items, the evaluation information G200 may also include information indicating whether or not it has RAG functionality.

[0060] Next, the flow of information processed by the recommendation system 100 will be explained with reference to Figure 6. Figure 6 is a diagram showing the flow of information in the recommendation system 100 according to the first embodiment.

[0061] When the task execution unit 110 receives request G11 from the user terminal 400, it causes the first language model 311 to generate a first response G21. The task execution unit 110 then supplies request G11 and the first response G21 to the recommendation information generation unit 130.

[0062] The evaluation information acquisition unit 120 acquires evaluation information G200 from the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit 130.

[0063] When the recommendation information generation unit 130 receives the request G11, the first response G21, and the evaluation information G200, it supplies this information to the second language model 312, causing the second language model 312 to generate recommendation information G22.

[0064] As described above, the recommendation system 100 causes the second language model 312, which has a different execution environment than the first language model 311, to generate recommendation information G22. The second language model 312 generates recommendation information G22 while referring to evaluation information G200.

[0065] Next, we will explain the variations in messages generated by the recommendation information generation unit 130 with reference to Figure 7. Figure 7 is a diagram showing the variations in messages generated by the recommendation information generation unit 130.

[0066] Figure 7 shows the information output by the recommendation information generation unit 130 upon receiving request G11, first response G21, and evaluation information G200. In addition to recommendation information G22, the recommendation information generation unit 130 generates upsell information G23. The upsell information G23 may include a message encouraging the user to use the recommended language model. Alternatively, the upsell information G23 may include a message prompting the user to choose whether or not to use the recommended language model.

[0067] In Figure 7, the upsell information G23 includes a message prompting the user to select either "Yes" or "No," along with the message, "Do you want to enable the recommended language model?" In this case, the recommendation system 100 provides the user terminal 400 with an interface to accept the operation to enable the recommended language model. This allows the recommendation system 100 to smoothly provide the user with the opportunity to enable the recommended language model. If the recommended language model is an optional paid service, the recommendation information generation unit 130 may generate a message including pricing information as the upsell information G23.

[0068] Furthermore, if the user terminal 400 selects to use the recommended language model, the task execution unit 110 may have the recommended language model generate a response to request G11. This allows the recommendation system 100 to provide the desired response while minimizing the number of steps required from the user.

[0069] The response system 10 and the recommendation system 100 have been described above. Each component of the recommendation system 100 may be implemented with dedicated hardware. In addition, some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be configured by a single chip or by multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the above-mentioned circuits, etc., and programs. In addition, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (field-programmable gate array), etc. can be used as the processor. Furthermore, at least some of the functions of this embodiment may be provided in the form of IaaS (Infrastructure as a Service), PaaS (Platform as a Service), or SaaS (Software as a Service).

[0070] Furthermore, the request G11 that the recommendation system 100 receives from the user terminal 400 may include data containing image data, audio data, or other information in addition to text data. Other data may include, for example, signals generated by a predetermined sensor. In this case, the first language model 311 and the second language model 312 are multimodal language models. A multimodal language model may be referred to as, for example, MMLLM (Multi Modal Large Language Model) or simply a multimodal model. A multimodal language model may generate a response using text data, or it may generate a response including image data or audio data. In this case, the evaluation information G200 may also include information indicating whether each language model is multimodal. The evaluation information G200 may also include information indicating what types of data in a multimodal environment each language model can handle. With this configuration, the recommendation system 100 can suitably propose an LLM that can appropriately respond to a multimodal request. In the following description, all language models may also be multimodal.

[0071] The above-described response system 10 has a configuration in which the recommendation system 100, database 200, and server 300 are each connected to the network N1 in a communicative manner, but the configuration of the response system 10 is not limited to the above. The response system 10 may be a single device in which at least a part of the recommendation system 100, database 200, and server 300 are integrated. Alternatively, the response system 10 may have a configuration in which, for example, a part of the recommendation system 100 is integrated with the database 200 or the server 300. In summary, according to this embodiment, it is possible to provide a recommendation system, recommendation method, and program that suitably propose an LLM that can appropriately respond to a user's request.

[0072] <Second Embodiment> Next, a second embodiment will be described. The recommendation system 100 according to the second embodiment differs from the recommendation system 100 described above in that it has an intent information generation unit 140.

[0073] Figure 8 is a block diagram of the recommendation system 100 according to the second embodiment. The recommendation system 100 mainly consists of a task execution unit 110, an evaluation information acquisition unit 120, a recommendation information generation unit 130, and an intent information generation unit 140.

[0074] The intent information generation unit 140, with the aim of supplementing request G11, causes the third language model 313, which has a different execution environment than the first language model 311, to generate intent information for input to the first language model 311 based on the request. The third language model 313 performs processing on the text data of the request, such as cleaning, tokenization, sentiment analysis, and semantic understanding. If the third language model 313 has RAG functionality, it may also perform a language search extracted from the text data of the request.

[0075] In this embodiment, the task execution unit 110 inputs at least a portion of the intent information to the first language model 311 to cause the first language model 311 to generate the first response G21. This allows the recommendation system 100 to improve the quality of the responses generated by the task execution unit 110.

[0076] Intent information includes, for example, the request type in a request received from the user terminal 400. That is, the intent information generation unit 140 generates intent information that includes information about the type of request from the context of the text data. Information about the type of request includes at least the request type.

[0077] The intent information may include information obtained by the third language model 313 by searching for language-related information contained in the request received from the user terminal 400. In other words, the intent information generation unit 140 may include a search engine that searches for information within a predetermined network in cooperation with the third language model 313, and may cause the third language model 313 and the search engine to generate intent information.

[0078] Figure 9 is a flowchart of the recommendation method according to the second embodiment. The flowchart shown in Figure 9 differs from the flowchart of the recommendation system 100 included in the sequence diagram shown in Figure 4 in that it has step S21 between step S11 and step S12.

[0079] In step S11, the task execution unit 110 receives a request G11 containing text data from the user terminal 400. The task execution unit 110 supplies the received request G11 to the intent information generation unit 140.

[0080] In step S21, the intent information generation unit 140 causes the third language model 313 to generate intent information based on the request G11 received from the task execution unit 110. The intent information generation unit 140 then supplies the intent information generated by the third language model 313 to the task execution unit 110.

[0081] In step S12, the task execution unit 110 supplies the request G11 and intent information to the first language model 311 to generate the first response G21.

[0082] In step S13, the evaluation information acquisition unit 120 acquires evaluation information G200 for each of the multiple language models from the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit 130.

[0083] In step S14, the recommendation information generation unit 130 works in cooperation with the server 300 to cause the second language model 312 to generate recommendation information G22. In this embodiment, the recommendation information generation unit 130 selects a recommended language model from multiple language models to generate a response to the request based on the request G11, intent information, first response G21, and evaluation information G200. Furthermore, the second language model 312 generates recommendation information G22 that includes a message presenting the recommended language model to the user.

[0084] Through the above-described process, the recommendation system 100 in this embodiment generates a first response G21 using intent information generated from request G11. This allows the recommendation system 100 to suppress a decrease in the generation quality of the first response G21. The recommendation system 100 also generates recommendation information G22 using intent information. This allows the recommendation system 100 to suppress a decrease in the generation quality of the recommendation information.

[0085] Next, with reference to Figure 10, the flow of information processed by the recommendation system 100 according to the second embodiment will be explained. Figure 10 is a diagram showing the flow of information in the recommendation system 100 according to the second embodiment.

[0086] When the task execution unit 110 receives request G11 from the user terminal 400, it supplies the received request G11 to the intent information generation unit 140. The intent information generation unit 140 supplies request G11 to the third language model 313, causing the third language model 313 to generate intent information G140. The intent information includes, for example, the request type, analysis data, and search words. The intent information generation unit 140 supplies the intent information G140 generated by the third language model 313 to the task execution unit 110.

[0087] The request type shown in Figure 10 is "summary generation." The request type for intent information G140 is the same as the request type for evaluation information G200. The analysis data may include, for example, information about tokenized words, information about language types such as Japanese and English, and sentiment analysis of the text. The search words are, for example, words extracted from request G11 that the third language model 313 determined needed to be searched. The search words may also include search results.

[0088] The task execution unit 110 supplies the request G11 and intent information G140 to the first language model 311, causing the first language model 311 to generate a first response G21. Upon receiving the first response G21 from the first language model 311, the task execution unit 110 supplies the received first response G21, the request G11, and the intent information G140 to the recommendation information generation unit 130.

[0089] The recommendation information generation unit 130 receives evaluation information G200 from the evaluation information acquisition unit 120. The recommendation information generation unit 130 also receives request G11, first response G21, and intent information G140 from the task execution unit 110. The recommendation information generation unit 130 supplies this received information to the second language model 312 to generate recommendation information G22.

[0090] The second embodiment has been described above. With the above configuration, the recommendation system 100 generates a first response G21 to the user's request G11 while suppressing a decrease in generation quality. The recommendation system 100 also presents the user with recommended language models, which are other language models that may have higher generation quality than the first language model 311. Therefore, according to this embodiment, it is possible to provide a recommendation system, recommendation method, and program that suitably propose an LLM that can appropriately respond to the user's request.

[0091] <Third Embodiment> Next, a third embodiment will be described. The third embodiment differs from the recommendation system 100 described above in that the recommendation system 100 has an evaluation unit 150.

[0092] Figure 11 is a block diagram of the recommendation system 100 according to the third embodiment. The recommendation system 100 according to this embodiment mainly consists of a task execution unit 110, an evaluation information acquisition unit 120, a recommendation information generation unit 130, an intent information generation unit 140, and an evaluation unit 150.

[0093] The evaluation unit 150 takes the request G11, the first response G21, and the intent information G140 as input and causes the fourth language model 314, which operates in a different execution environment than the first language model 311, to output an evaluation score for the first response G21 generated by the first language model 311. The evaluation score is an indicator of the quality of generation. Here, as described above, the quality of generation may include elements such as the accuracy, relevance, consistency, and customizability of the response. In other words, the fourth language model 314 is configured to generate an evaluation score based on these elements.

[0094] The evaluation score is represented, for example, by an integer from 0 to 100. In this case, for example, a higher value may be defined as higher production quality. However, the evaluation score is not limited to the above definition. The evaluation score may be quantitative or a qualitative indicator such as "good" or "bad".

[0095] In this embodiment, the recommendation information generation unit 130 generates recommendation information G22 by taking into account the evaluation score. That is, the recommendation information generation unit 130 in this embodiment generates recommendation information G22 by adding at least the evaluation score generated by the evaluation unit 150 to the second language model 312. As a result, the recommendation system 100 can generate recommendation information G22 in the second language model 312 in a manner that makes it easy to compare with the evaluation information G200 stored in the database 200.

[0096] The evaluation unit 150 in this embodiment may have a function to supply request G11 and the evaluation score corresponding to request G11 to a database for storage in evaluation information. With this configuration, the response system 10 can enrich the evaluation information G200 by the user using the response system 10.

[0097] Next, with reference to Figure 12, the processes performed by the recommendation system 100 according to this embodiment will be described. Figure 12 is a flowchart of the recommendation method according to the third embodiment. The flowchart shown in Figure 12 differs from the flowchart shown in Figure 9 in that it has step S31 after step S12.

[0098] In step S12 of this embodiment, the task execution unit 110 supplies the request G11 and intent information G140 to the first language model 311 to generate the first response G21. The task execution unit 110 supplies the request G11, the first response G21 generated by the first language model 311, and the intent information G140 to the evaluation unit 150.

[0099] In step S31, the evaluation unit 150 supplies the received request G11, first response G21, and intent information G140 to the fourth language model 314, causing the fourth language model 314 to generate an evaluation score. The evaluation unit 150 then supplies the evaluation score generated by the fourth language model 314 to the recommendation information generation unit 130.

[0100] In step S13, the evaluation information acquisition unit 120 acquires evaluation information G200 from the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit 130.

[0101] In step S14, the recommendation information generation unit 130 works in cooperation with the server 300 to cause the second language model 312 to generate recommendation information G22. In this embodiment, the recommendation information generation unit 130 selects a recommended language model from a plurality of language models to generate a response based on the request G11, intent information G140, first response G21, evaluation information G200, and evaluation score generated by the evaluation unit 150. Furthermore, the second language model 312 generates recommendation information G22 that includes a message presenting the recommended language model to the user.

[0102] (Evaluation information G200) Next, the evaluation information G200 according to this embodiment will be described with reference to Figure 13. Figure 13 is a second figure showing the evaluation information G200. The evaluation information G200 shown in Figure 13 differs from the evaluation information G200 shown in Figure 5 in that it includes a score. The score is an evaluation of the performance response to the performance prompt.

[0103] For example, evaluation information G200 includes the following information: The first piece of information states that when user B sent the request "Please summarize text D" to language model "Model 1-ver.3", the actual response received was "I will summarize...", and the score of this actual response was 65. The second piece of information states that when user B sent the request "Please summarize text D" to language model "Model 2-ver.2", the actual response received was "The following is the summarized text...", and the score of this actual response was 39.

[0104] The third piece of information includes the fact that when user C sent a request to the language model "Model 1-ver.3" saying "Please generate code to execute...", the code generated by this language model was obtained as the actual response. The third piece of information also includes the fact that the score of this actual response was 52.

[0105] The fourth piece of information includes the fact that when user C sent a request to the language model "Model 1-ver.2" saying "Please generate code to execute...", the code generated by this language model was obtained as the actual response. The fourth piece of information also includes the fact that the score of this actual response was 48.

[0106] By including the information described above, the recommendation information generation unit 130 supplies evaluation information G200, including the score, to the second language model 312. This allows the second language model 312 to compare the evaluation score G150 corresponding to the first response G21 with the evaluation information G200. Therefore, the recommendation information generation unit 130 can cause the second language model 312 to suitably generate recommendation information, including the recommended language model, based on objective evaluation.

[0107] Next, the flow of information processed by the recommendation system 100 according to this embodiment will be described with reference to Figure 14. Figure 14 is a diagram showing the flow of information in the recommendation system 100 according to the third embodiment.

[0108] In this embodiment, the task execution unit 110 uses the intent information G140 received from the intent information generation unit 140 to cause the first language model 311 to generate a first response G21. The task execution unit 110 supplies the request G11, the first response G21, and the intent information G140 to the evaluation unit 150.

[0109] The evaluation unit 150 supplies this information received from the task execution unit 110 to the fourth language model 314. This causes the evaluation unit 150 to cause the fourth language model 314 to generate an evaluation score G150. In the example shown in Figure 14, the evaluation score G150 is shown as "Score: 65". That is, the evaluation score G150 is 65. Once the evaluation unit 150 has caused the fourth language model 314 to generate the evaluation score, it supplies the request G11, the first response G21, the intent information G140, and the evaluation score G150 to the recommendation information generation unit 130.

[0110] The evaluation unit 150 also supplies additional information G151 to the database 200. The additional information G151 is supplied to the database 200 for the purpose of being added to the evaluation information G200. In other words, the additional information G151 is information corresponding to the evaluation information G200. The additional information G151 includes at least the corresponding request G11, first response G21, and evaluation score G150. The additional information G151 may also include a part of the intent information G140.

[0111] In this embodiment, the recommendation information generation unit 130 receives the request G11, the first response G21, the intent information G140, and the evaluation score G150 from the evaluation unit 150, and also receives the evaluation information G200 from the evaluation information acquisition unit 120. The recommendation information generation unit 130 causes the fourth language model 314 to generate recommendation information G22 that takes the evaluation score G150 into account.

[0112] The third embodiment has been described above. The recommendation system 100 according to this embodiment can select a more suitable recommended language model by using the evaluation score G150. Furthermore, the recommendation system 100 according to this embodiment can efficiently store evaluation information G200. By using the efficiently stored evaluation information G200, the recommendation system 100 can select an even more suitable recommended language model. Therefore, according to this embodiment, it is possible to provide a recommendation system, recommendation method, and program that suitably propose an LLM that can appropriately respond to a user's request.

[0113] <Fourth Embodiment> Next, a fourth embodiment will be described. In the fourth embodiment, the function of the recommendation information generation unit 130 differs from that of the recommendation system 100 described above. That is, the recommendation information generation unit 130 according to this embodiment outputs recommendation information, including information regarding the difference between the first language model 311 and the recommended language model, to the user terminal 400 used by the user who received the request G11.

[0114] Figure 15 is a flowchart of the recommendation method according to the fourth embodiment. The flowchart shown in Figure 15 differs from the flowchart shown in Figure 12 in that it includes the processes of steps S141 to S144 instead of the process of step S14.

[0115] In step S141, the recommendation information generation unit 130 causes the second language model 312 to select a recommended language model based on the evaluation score G150.

[0116] In step S142, the recommendation information generation unit 130 supplies request G11 to the recommended language model selected by the second language model 312, causing it to generate a response to request G11.

[0117] In step S143, the recommendation information generation unit 130 causes the second language model 312 to compare the response from the recommended language model with the first response G21 from the first language model 311.

[0118] In step S144, the recommendation information generation unit 130 causes the second language model 312 to generate recommendation information using the results of the comparison described above.

[0119] Referring to Figure 16, an example of recommendation information generated by the recommendation information generation unit 130 according to the fourth embodiment will be described. Figure 16 is a diagram showing the recommendation information according to the fourth embodiment. The recommendation information generation unit 130 shown in Figure 16 outputs first explanatory information G31, second explanatory information G32, and upsell information G23 as recommendation information.

[0120] The first explanatory information G31 includes a message that compares the recommended language model with the first language model 311 and explains the advantages of the recommended language model. The recommendation information generation unit 130 causes the second language model 312 to generate the first explanatory information G31 and outputs the generated first explanatory information G31 to the user terminal 400. The first explanatory information G31 shown in Figure 16 includes the message, "Model 1-ver.3 is superior to Model 2-ver.2 in the following respects: (1)..., (2)...."

[0121] By outputting the first explanatory information G31, the recommendation system 100 can clearly demonstrate the advantages of the recommended language model to the user. Furthermore, by outputting the first explanatory information G31, the recommendation system 100 can effectively encourage the user to use the recommended language model.

[0122] The second explanatory information G32 compares the response from the recommended language model with the first response G21 from the first language model 311 and includes a message explaining the advantages of the recommended language model's response. The recommendation information generation unit 130 causes the second language model 312 to generate the second explanatory information G32 and outputs the generated second explanatory information G32 to the user terminal 400. The second explanatory information G32 shown in Figure 16 includes the message, "Using model 1-ver.3 for this request will improve (3)... and (4)...."

[0123] By outputting the second explanatory information G32, the recommendation system 100 can clearly indicate to the user the specific improvements that can be made by using the recommended language model. Furthermore, by outputting the first explanatory information G31, the recommendation system 100 can effectively encourage the user to use the recommended language model.

[0124] The recommendation information generation unit 130 can output upsell information G23 in addition to the first explanatory information G31 and second explanatory information G32 described above. By outputting recommendation information in this combination, the recommendation system 100 can smoothly encourage users to use the recommended language model.

[0125] The fourth embodiment has now been described. In steps S141 to S144 of the flowchart in Figure 15, the recommendation information generation unit 130 may supply at least a portion of the intent information G140 to the recommended language model. This allows the recommendation information generation unit 130 to input prompts similar to those of the first language model 311 to the recommended language model.

[0126] As described above, according to the fourth embodiment, the recommendation system 100 can suitably encourage users to use language models with high generation quality.

[0127] (Variation of response system 10) Next, we will describe variations in the usage of the response system 10. Figure 17 is a block diagram showing variations of the response system 10. The response system 10 shown in Figure 17 differs from the response system 10 shown in Figure 1 in that it connects to the operator's equipment 500 in a communicative manner instead of the user terminal 400.

[0128] The business device 500 is a computer, server, cloud service, PaaS or SaaS, etc., for providing business-related services to multiple customers. The business device 500 connects to multiple customer terminals 600 via any means of communication and provides the business's services. Here, the services that the business provides to customers include the response service provided by the response system 10. That is, the response system 10 provides the response service to the business's customers via the business device 500 and also provides the functions of the recommendation system 100. With the above configuration, the response system 10 can provide the functions of this disclosure to various businesses.

[0129] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0130] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate. [Explanation of Symbols]

[0131] 10 Answer System 100 Recommendation Systems 110 Task Execution Unit 120 Evaluation Information Acquisition Unit 130 Recommendation Information Generation Unit 140 Intent Information Generation Unit 150 Evaluation Department 200 databases 300 servers 311 First Language Model 312 Second Language Model 313 Third Language Model 314. Fourth Language Model 400 user terminals 500 Operator equipment 600 customer terminals 1000 computers 1010 Bus 1020 processor 1030 memory 1040 Storage Devices 1050 Input / Output Interface 1060 Network Interfaces G11 Request G21 1st answer G22 Recommendation Information G23 Upsell Information G31 First Explanatory Information G32 Second Explanation Information G140 Intent Information G150 Rating Score G151 Additional Information G200 Evaluation Information N1 Network

Claims

1. A task execution unit that receives requests from users that include text data, An evaluation information acquisition unit that obtains evaluation information corresponding to the request type for each of multiple language models from a predetermined database, A recommendation information generation unit causes a language model to select a recommended language model from among the multiple language models to generate a response to the request based on the request and the evaluation information. The system includes an intent information generation unit that causes a language model to generate intent information to supplement the request based on the request type of the request, The task execution unit inputs at least a portion of the intent information into the language model and causes the recommended language model to generate a response to the request. Answer system.

2. In the answer system described in claim 1, The intent information generation unit generates intent information, including information about the type of request, from the context of the text data. Answer system.

3. In the answer system described in claim 1, The intent information generation unit includes a search engine that searches for information within a predetermined network in cooperation with a language model, and causes the language model and the search engine to generate the intent information. Answer system.

4. In the answer system described in claim 1, The system further includes an evaluation unit that takes the aforementioned request, the aforementioned response, and the aforementioned intent information as input and causes the language model to output an evaluation score for the response generated by the language model, The recommendation information generation unit generates recommendation information that includes a message presenting the recommended language model to the user, taking into account the evaluation score. Answer system.

5. In the response system described in claim 4, The evaluation unit supplies the request and the evaluation score corresponding to the request to the database in order to store them in the evaluation information. Answer system.

6. Computers We accept requests from users that include text data. Evaluation information corresponding to the request type for each of the multiple language models is obtained from a designated database. Based on the request and the evaluation information, the language model is instructed to select a recommended language model from the multiple language models to generate a response to the request. Based on the request type of the aforementioned request, the language model generates intent information to supplement the request. At least a portion of the intent information is input into the language model to cause the recommended language model to generate a response to the request. How to answer.

7. We accept requests from users that include text data. Evaluation information corresponding to the request type for each of the multiple language models is obtained from a designated database. Based on the request and the evaluation information, the language model is instructed to select a recommended language model from the multiple language models to generate a response to the request. Based on the request type of the aforementioned request, the language model generates intent information to supplement the request. At least a portion of the intent information is input into the language model to cause the recommended language model to generate a response to the request. The computer will execute the answer method. program.

Images