Answering system, answering method and program

Abstract

To provide a recommendation system or the like for suitably proposing an LLM capable of appropriately answering a user's request.SOLUTION: In a recommendation system, a task execution unit receives a request including text data from a user and causes a first language model, which is one language model designated by the user from a plurality of language models, to generate a first answer to the request. The evaluation information acquisition unit acquires evaluation information on each of the plurality of language models from a predetermined database. The recommendation information generation unit causes a second language model having an execution environment different from that of the first language model to select a recommended language model for generating an answer to the request from the plurality of language models on the basis of the request, the first answer, and the evaluation information. Further, the recommendation information generation unit causes the second language model to generate recommendation information including a message for presenting the recommended language model to the user.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a recommendation system, a recommendation method, and a program. [Background technology]

[0002] Generative AI (Artificial Intelligence) services that use large language models (LLMs) to generate answers in response to various user requests are becoming popular. Users can choose from multiple generative AI services. In addition, users can select any of multiple LLMs with different characteristics and submit requests to the generative AI service. In this regard, several proposals have been disclosed for selectively using multiple trained models.

[0003] For example, the technology described in Patent Document 1 learns the life logs of individual experts and stores a learning model that predicts each individual expert's answer as a digital clone, inputs question data into the corresponding digital clone, outputs the predicted answer, and evaluates multiple digital clones.

[0004] Furthermore, the technology described in Patent Document 2 selects one of a plurality of learning models according to the content of the acquired question and the request of the questioner, and inputs the question data into the selected learning model.

[0005] The system described in Patent Document 3 includes a speech circuit that automatically speaks in response to the content of the operator's speech, a circuit that stores and selects different statistical language models according to multiple situations, and a speech recognition circuit that recognizes the content of the speech after it is spoken. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2023-114460 [Patent Document 2] Japanese Patent Publication No. 2023-082970 [Patent Document 3] Japanese Patent Application Publication No. 07-104786 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned techniques can only be used for specific learning models. On the other hand, there is a need to develop a technique to select an LLM that can appropriately respond to a user request.

[0008] In view of the above-mentioned problems, the present disclosure aims to provide a recommendation system etc. that suitably suggests LLMs that can appropriately respond to user requests. [Means for solving the problem]

[0009] A recommendation system according to the present disclosure includes a task execution unit, a rating information acquisition unit, and a recommendation information generation unit. The task execution unit receives a request including text data from a user, and causes a first language model, which is one language model specified by the user from among multiple language models, to generate a first response to the request. The rating information acquisition unit acquires rating information for each of the multiple language models from a predetermined database. The recommendation information generation unit causes a second language model, having an execution environment different from that of the first language model, to select a recommended language model for generating a response to the request from the multiple language models based on the request, the first response, and the rating information. Furthermore, the recommendation information generation unit causes the second language model to generate recommendation information including a message presenting the recommended language model to the user.

[0010] In a recommendation method according to the present disclosure, a computer executes the following processes. The computer receives a request including text data from a user, and generates a first response to the request in a first language model, which is one language model specified by the user from among multiple language models. The computer acquires evaluation information for each of the multiple language models from a predetermined database. The computer selects a recommended language model for generating a response to the request from the multiple language models based on the request, the first response, and the evaluation information, in a second language model having an execution environment different from that of the first language model. The computer generates recommendation information including a message presenting the recommended language model to the user in the second language model.

[0011] A program according to the present disclosure causes a computer to execute the following recommendation method. The computer receives a request including text data from a user, and generates a first response to the request in a first language model, which is one language model specified by the user from among multiple language models. The computer acquires evaluation information for each of the multiple language models from a predetermined database. The computer selects a recommended language model for generating a response to the request from the multiple language models based on the request, the first response, and the evaluation information, in a second language model having an execution environment different from that of the first language model. The computer causes the second language model to generate recommendation information including a message presenting the recommended language model to the user. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to provide a recommendation system, a recommendation method, and a program that suitably propose an LLM that can appropriately respond to a user request. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of a response system according to a first embodiment. [Figure 2]FIG. 1 is a block diagram of a recommendation system according to a first embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. [Figure 4] 1 is a flowchart of a recommendation method according to the first embodiment. [Figure 5] FIG. 1 is a first diagram showing evaluation information. [Figure 6] FIG. 2 is a diagram showing the flow of information in the recommendation system according to the first embodiment. [Figure 7] 10A and 10B are diagrams illustrating variations of messages generated by a recommendation information generating unit. [Figure 8] FIG. 10 is a block diagram of a recommendation system according to a second embodiment. [Figure 9] 10 is a flowchart of a recommendation method according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing the flow of information in a recommendation system according to a second embodiment. [Figure 11] FIG. 10 is a block diagram of a recommendation system according to a third embodiment. [Figure 12] 10 is a flowchart of a recommendation method according to a third embodiment. [Figure 13] FIG. 2 is a second diagram showing evaluation information. [Figure 14] FIG. 10 is a diagram showing the flow of information in a recommendation system according to a third embodiment. [Figure 15] 10 is a flowchart of a recommendation method according to a fourth embodiment. [Figure 16] 10A and 10B are diagrams illustrating variations of messages generated by a recommendation information generating unit. [Figure 17] FIG. 10 is a block diagram showing a variation of the answering system. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0015] <First Embodiment> (Answer System 10) The answer system 10 will be described with reference to FIG. 1. FIG. 1 is a block diagram of the answer system 10 according to the first embodiment. The answer system 10 receives a request from a user, selects a recommended language model for generating an answer to the received request from a plurality of language models, and presents recommendation information related to the selected recommended language model to the user. The answer system 10 is communicably connected to a user terminal 400 via a network N1. The answer system 10 mainly includes a recommendation system 100, a database 200, and a server 300.

[0016] The recommendation system 100 is a computer or server having a communication function. The recommendation system 100 is communicably connected to a user terminal 400 via a network N1. The recommendation system 100 is also communicably connected to a database 200 and a server 300 via the network N1. As a result, the recommendation system 100 cooperates with the database 200 and the server 300 to select the above-mentioned recommended language model.

[0017] Furthermore, the recommendation system 100 receives a request from the user terminal 400, as well as a specification of a language model for generating an answer. In this case, the recommendation system 100 causes the specified language model to generate an answer to the request. Details of the recommendation system 100 will be described later.

[0018] The database 200 is a computer, server, or storage device including at least a non-volatile memory. The database 200 is communicatively connected to the recommendation system 100 via a network N1. The database 200 stores at least evaluation information G200. The evaluation information G200 is information evaluating each of a plurality of language models that can be used by the recommendation system 100. The evaluation information G200 is used when the recommendation system 100 selects a language model to recommend. The database 200 supplies at least a portion of the evaluation information G200 to the recommendation system 100 in response to a request from the recommendation system 100.

[0019] The server 300 is a server that is communicatively connected to the recommendation system 100. The server 300 includes a plurality of different language models. Specifically, for example, the server 300 has a first language model 311, a second language model 312, a third language model 313, a fourth language model 314, and the like. These language models are LLMs and are configured to be usable by the recommendation system 100. In other words, the language models included in the server 300 accept access from the recommendation system 100, generate a response to an arbitrary request, and provide the generated response to the recommendation system 100.

[0020] The language model possessed by the server 300 may have all the functions of a generative AI in the server 300. The language model possessed by the server 300 may also have an API (Application Programming Interface) for causing a predetermined LLM to function in the server 300. In this case, the language model may be configured to function as a generative AI by linking with an external device.

[0021] The multiple different language models possessed by server 300 may be, for example, language models provided by multiple different generative AI service providers. The multiple different language models may also be language models provided by the same generative AI service, but different versions of each other. These language models may also have a function called RAG (Retrieval-Augmented Generation). RAG is a mechanism that searches for external information during text generation by the LLM and uses the search results to generate an answer. RAG is also called "retrieval augmented generation" or "retrieval augmented generation." Having RAG in the LLM allows the LLM to generate information that is highly relevant to the request.

[0022] The user terminal 400 is a computer, a smartphone, a tablet terminal, or the like used by a user. An application for accessing the response system 10 is installed in the user terminal 400. A user using the user terminal 400 sends a request to the response system 10 via this application. A user using the user terminal 400 also receives a response to the request from the response system 10 via this application. Note that although there is one user terminal 400 in FIG. 1, the response system 10 can be connected to multiple user terminals 400 so that they can communicate with each other. The application for accessing the answering system 10 is, for example, a web browser.

[0023] (Recommendation System 100) 2 is a block diagram of a recommendation system according to Embodiment 1. The recommendation system 100 mainly includes a task execution unit 110, a rating information acquisition unit 120, and a recommendation information generation unit .

[0024] The task execution unit 110 receives a request including text data from a user, and generates a first answer to the request in a first language model 311, which is one language model designated by the user from among multiple language models. That is, the task execution unit 110 cooperates with the server 300 to input the request received from the user to the first language model 311 designated by the user, and causes the first language model 311 to generate a first answer. In this case, the request received from the user may be referred to as a prompt. In the present disclosure, a "prompt" is an instruction sentence that is input into a language model to obtain an answer. The task execution unit 110 may present an answer to the request received from the user on the user terminal 400.

[0025] The evaluation information acquisition unit 120 acquires evaluation information for each of the multiple language models from a predetermined database. In this case, the predetermined database is the database 200. The evaluation information G200 stored in the database 200 includes evaluations for the first language model 311. The evaluations for the first language model 311 include performance prompts, which are information including at least a portion of prompts previously input to the first language model 311, and information on evaluations of performance answers, which are answers generated by the first language model 311 in response to these performance prompts. This allows the recommendation system 100 to refer to how performance answers to performance prompts received by the first language model 311 have been evaluated. The recommendation system 100 can also refer to information on similar evaluations in language models other than the first language model 311.

[0026] The recommendation information generation unit 130 selects a recommended language model for generating a response to the request from a plurality of language models based on the request, the first response, and the evaluation information G200. The recommendation information generation unit 130 also causes a second language model 312, which has an execution environment different from that of the first language model 311, to generate recommendation information including a message presenting the recommended language model to the user.

[0027] More specifically, the recommendation information generation unit 130 inputs, for example, a request received from a user and a first answer generated by the first language model 311 to the second language model 312. The second language model 312 is configured to select a recommended language model by referring to the input request and first answer and the evaluation information G200, and to generate a message presenting the selected recommended language model to the user.

[0028] In the present disclosure, "language models with different execution environments" includes cases where the types of language models themselves are different. Furthermore, "language models with different execution environments" includes cases where the type of language model is the same but the versions are different. Furthermore, "language models with different execution environments" includes cases where the type and version of the language model are the same but the conditions instructed when generating an answer are different. The conditions instructed when generating an answer are, for example, the content of prompt instructions. In this case, the content of prompt instructions may include instruction guidelines such as "please summarize," "please suggest an idea," or "please rate the text." Such instruction guidelines may also be referred to as "request types."

[0029] As described above, the recommendation information generation unit 130 causes the second language model 312 to generate recommendation information. If the recommended language model included in the recommendation information is a language model different from the first language model 311, the recommendation system 100 presents to the user a recommended language model that is likely to generate a more appropriate answer to the user's request than the first language model 311. In this case, the user who receives the recommendation information can specify a recommended language model and send a request to the recommendation system 100. This allows the recommendation system 100 to provide an appropriate answer to the user.

[0030] In addition, if the recommended language model included in the recommendation information is the first language model 311, the recommendation system 100 may present the user with a message indicating that the first language model 311 may generate an appropriate answer to the user's request.

[0031] In the above-described recommendation system 100, the language model that the user can specify may be set differently from the recommended language model that the recommendation information generator 130 can select. This is the case, for example, when language models that the user can additionally use, for a fee or free of charge, are set depending on the type of service the user receives. In this case, for example, the user can use the recommended language model as an optional service. That is, the task execution unit 110 accepts the specification of the first language model 311 from among multiple language models that the user can specify, and the recommendation information generator 130 selects a recommended language model from multiple language models that exceed the range that the user can specify.

[0032] With this configuration, the recommendation system 100 can suitably present optional recommended language models to users. Furthermore, users of the recommendation system 100 can select additional recommended language models as needed in actual usage situations.

[0033] The recommended language model may be set as a paid option. In this case, the recommendation system 100 may prompt the user to select whether or not to use the recommended language model as a paid optional service.

[0034] In generative AI services using LLMs, users desire more appropriate answers to their desired requests. Using an appropriate LLM according to the request increases the likelihood that the user will receive an answer of higher quality. Here, generation quality can include factors such as the accuracy, relevance, consistency, and customizability of the answer. Accuracy is a factor indicating whether the answer is based on facts. Relevance is a factor indicating whether the answer is directly related to the user's question. Consistency is a factor indicating whether the answer is logical and has consistent content. Customizability is a factor indicating whether the answer is tailored to the user's individual needs and background.

[0035] For example, suppose a user sends a request such as "What's the weather in Tokyo today?" and the first language model 311 specified by the user does not have the RAG function. In this case, the first language model 311 generates a response such as "It is not possible to check the weather in Tokyo today." On the other hand, suppose the recommended language model has the RAG function. In this case, the recommended language model searches for the weather in Tokyo, for example, via an Internet search, and generates a response. The recommendation information generation unit 130 presents the user with a recommended language model with the RAG function. This allows the recommendation system 100 to present the user with a generation AI service with high generation quality. Furthermore, if the provider of the response system 10 offers users a language model with high generation quality as an option, it is expected to expand upselling opportunities.

[0036] (Example of hardware configuration) Fig. 3 is a block diagram illustrating an example of the hardware configuration of a computer. The recommendation system 100 described above may have the configuration shown in Fig. 3. The computer 1000 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0037] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0038] The processor 1020 is a circuit including an arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0039] The memory 1030 is a main storage device realized using a RAM (Random Access Memory) or the like.

[0040] The storage device 1040 is an auxiliary storage device such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or a read only memory (ROM), etc. The storage device 1040 stores programs for realizing the functions of the present disclosure.

[0041] The processor 1020 reads the program into the memory 1030 and executes it, thereby causing the processor 1020 to execute the functions corresponding to the program. In other words, the program stored in the memory 1030 causes the computer 1000 to execute the functions of the present disclosure.

[0042] The input / output interface 1050 connects the computer 1000 to a predetermined input / output device. The input / output device is, for example, an input device such as a keyboard, an output device such as a display, or an input / output device in which a touch panel is superimposed on a display.

[0043] The network interface 1060 is an interface for connecting the computer 1000 to a predetermined communication network.

[0044] (Recommendation method) Next, the processing executed by the recommendation system 100 will be described with reference to Fig. 4. Fig. 4 is a sequence diagram showing the recommendation method according to the first embodiment. The sequence diagram in Fig. 4 includes information exchange between the user terminal 400 and the recommendation system 100, and the recommendation method executed by the recommendation system 100. In the recommendation method according to the present disclosure, the recommendation system 100 executes the following processing from step S10 to step S14.

[0045] In step S10, the recommendation system 100 receives a specification of a language model from the user terminal 400.

[0046] In step S11, the task execution unit 110 receives a request including text data from the user terminal 400. In the example shown in Fig. 4, the user terminal 400 transmits a message to the recommendation system 100 as a request G11, saying "Please summarize the following sentence D. Sentence D:..."

[0047] In step S12, the task executing unit 110 causes the first language model 311, which is one language model specified by the user from multiple language models, to generate a first answer to the received request. In this case, the task executing unit 110 cooperates with the server 300 to cause the first language model 311 to generate an answer. That is, the task executing unit 110 supplies the request G11 received from the user terminal 400 to the specified first language model 311. The task executing unit 110 receives the message "Summarize sentence D:..." as the first answer G21 from the first language model 311. Upon receiving the first answer G21, the task executing unit 110 transmits the received first answer G21 to the user terminal 400.

[0048] In step S13, the evaluation information acquisition unit 120 acquires evaluation information G200 related to each of the plurality of language models from a predetermined database, that is, the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit 130.

[0049] In step S14, the recommendation information generator 130, in cooperation with the server 300, causes the second language model 312, which has an execution environment different from that of the first language model 311, to generate recommendation information G22. In this case, the second language model 312 selects a recommended language model for generating a response to the request from multiple language models based on the request G11, the first answer G21, and the evaluation information G200. Furthermore, the second language model 312 generates recommendation information G22 including a message presenting the recommended language model to the user.

[0050] The above has described the processing executed by the recommendation system 100. In the above processing, the recommendation system 100 may determine that the recommended language model matches the first language model 311. When the recommended language model matches the first language model 311, the recommendation information generation unit 130 may refrain from transmitting the generated recommendation information to the user terminal 400. By performing such processing, the recommendation system 100 can prevent redundant information from being presented to the user. Note that the language model specified by the user in step S10 may be set in advance.

[0051] (Evaluation Information G200) Next, the evaluation information G200 will be described with reference to Fig. 5. Fig. 5 is a first diagram showing the evaluation information G200. The evaluation information G200 includes a user name, a performance prompt, a performance answer, a request type, and a language model.

[0052] The user name is a name for identifying the user who operates the user terminal 400. The user name may be a predetermined ID (Identifier), or may be the user's name or surname.

[0053] An achievement prompt is input information that includes at least a part of a request entered by a user. An achievement answer is an answer generated by a language model in response to a request. A request type is a broad classification of the type of answer that a language model is to generate by an achievement prompt. A request type can also be said to be the content of an instruction by an achievement prompt.

[0054] The request types include, for example, generating a summary, generating code, generating a translation, assisting in generating ideas, presenting search results for information, presenting example sentences, etc. Any definition can be set for the request type.

[0055] The language model is information that can identify the language model. The language model may be a unique identifier, a name of the language model, or the like.

[0056] Specifically, for example, the evaluation information G200 includes the following information: The first information is that when user B sent a request "Please summarize sentence D" to language model "Model 1-ver. 3," he received an actual response of "I will summarize it..." The second information is that when user B sent a request "Please summarize sentence D" to language model "Model 2-ver. 2," he received an actual response of "Below is the summarized sentence..."

[0057] The third information includes the content that when user C sent a request to the language model “Model 1-ver.3” saying “Please generate code that executes...”, the code generated by this language model was obtained as the actual answer.

[0058] The fourth information includes the content that when user C sent a request to the language model “Model 1-ver.2” saying “Please generate code that executes...”, the code generated by this language model was obtained as the actual answer.

[0059] By including the above information, the recommendation information generation unit 130 causes the second language model to generate recommendation information including the recommended language model. Note that the evaluation information G200 may include some of the above items. For example, the evaluation information G200 may not include a user name. In addition to the above items, the evaluation information G200 may also include, for example, information indicating whether or not the device has a RAG function.

[0060] Next, the flow of information processed by the recommendation system 100 will be described with reference to Fig. 6. Fig. 6 is a diagram showing the flow of information in the recommendation system 100 according to the first embodiment.

[0061] When the task execution unit 110 receives the request G11 from the user terminal 400, it causes the first language model 311 to generate a first answer G21. The task execution unit 110 supplies the request G11 and the first answer G21 to the recommendation information generation unit .

[0062] The evaluation information acquisition unit 120 acquires the evaluation information G200 from the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit .

[0063] Upon receiving the request G11, the first response G21, and the evaluation information G200, the recommendation information generator 130 supplies the received information to the second language model 312, causing the second language model 312 to generate recommendation information G22.

[0064] As described above, the recommendation system 100 causes the second language model 312, which has an execution environment different from that of the first language model 311, to generate the recommendation information G22. The second language model 312 generates the recommendation information G22 while referring to the evaluation information G200.

[0065] Next, variations of messages generated by the recommendation information generating unit 130 will be described with reference to Fig. 7. Fig. 7 is a diagram showing variations of messages generated by the recommendation information generating unit 130.

[0066] 7 shows information output by the recommendation information generation unit 130 that has received the request G11, the first response G21, and the evaluation information G200. The recommendation information generation unit 130 generates upsell information G23 in addition to the recommendation information G22. The upsell information G23 may include a message that prompts the user to use the recommended language model. Alternatively, the upsell information G23 may include a message that prompts the user to select whether or not to use the recommended language model.

[0067] In FIG. 7, the upsell information G23 includes a message asking "Do you want to enable the recommended language model?" and prompting the user to select either "Yes" or "No." In this case, the recommendation system 100 provides the user terminal 400 with an interface for accepting an operation to enable the recommended language model. This allows the recommendation system 100 to provide the user with an opportunity to smoothly enable the recommended language model. If the recommended language model is an optional paid service, the recommendation information generation unit 130 may generate a message including fee information as the upsell information G23.

[0068] When an operation to select use of a recommended language model is received from the user terminal 400, the task executing unit 110 may cause the recommended language model to generate a response to the request G11. This allows the recommendation system 100 to provide a desired response while reducing the number of steps required by the user.

[0069] The answer system 10 and the recommendation system 100 have been described above. Each component of the recommendation system 100 may be realized by dedicated hardware. Furthermore, some or all of the components may be realized by general-purpose or dedicated circuits, processors, etc., or a combination thereof. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components may be realized by a combination of the above-mentioned circuits, etc., and a program. Furthermore, a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), etc. may be used as the processor. Furthermore, at least some of the functions of this embodiment may be provided in the form of infrastructure as a service (IaaS), platform as a service (PaaS), software as a service (SaaS), etc.

[0070] Furthermore, the request G11 received by the recommendation system 100 from the user terminal 400 may be data including image data, audio data, or other information in addition to text data. Examples of data including other information include signals generated by a specific sensor. In this case, the first language model 311 and the second language model 312 are multimodal language models. A multimodal language model may be referred to as an MMLLM (Multi Modal Large Language Model) or a multimodal model. A multimodal language model may generate a response using text data or may generate a response including image data or audio data. In this case, the evaluation information G200 may include information indicating whether each language model is multimodal. The evaluation information G200 may include information indicating which type of multimodal data each language model can handle. With this configuration, the recommendation system 100 can suitably propose an LLM that can appropriately respond to a multimodal request. Note that in the following description, all language models may be multimodal.

[0071] The above-described answer system 10 has a configuration in which the recommendation system 100, the database 200, and the server 300 are each communicably connected to the network N1, but the configuration of the answer system 10 is not limited to that described above. The answer system 10 may be an apparatus in which at least a portion of the recommendation system 100, the database 200, and the server 300 are integrated. The answer system 10 may also be configured, for example, in which a portion of the recommendation system 100 is integrated with the database 200 or the server 300. As described above, according to the present embodiment, it is possible to provide a recommendation system, a recommendation method, and a program that suitably propose an LLM that can appropriately answer a user request.

[0072] Second Embodiment Next, a description will be given of a second embodiment. The recommendation system 100 according to the second embodiment differs from the recommendation system 100 described above in that it includes an intention information generation unit 140.

[0073] 8 is a block diagram of a recommendation system 100 according to the second embodiment. The recommendation system 100 mainly includes a task execution unit 110, an evaluation information acquisition unit 120, a recommendation information generation unit 130, and an intention information generation unit 140.

[0074] The intention information generation unit 140 generates, based on the request, intention information to be input to the first language model 311 in a third language model 313 having an execution environment different from that of the first language model 311, for the purpose of supplementing the request G11. The third language model 313 performs processing such as cleaning, tokenization, sentiment analysis, and semantic understanding on the text data that is the request. If the third language model 313 has a RAG function, it may search for a language extracted from the text data that is the request.

[0075] The task executing unit 110 in this embodiment inputs at least a part of the intention information into the first language model 311 to generate a first answer G21 in the first language model 311. This allows the recommendation system 100 to improve the quality of the answer generated by the task executing unit 110.

[0076] The intention information includes, for example, the request type of the request received from the user terminal 400. That is, the intention information generation unit 140 generates intention information including information about the type of request from the context of the text data. The information about the type of request is information that includes at least the request type.

[0077] The intention information may include information acquired by the third language model 313 by searching for information related to the language included in the request received from the user terminal 400. In other words, the intention information generation unit 140 may include a search engine that searches for information within a predetermined network in cooperation with the third language model 313, and may cause the third language model 313 and the search engine to generate the intention information.

[0078] Fig. 9 is a flowchart of a recommendation method according to the second embodiment. The flowchart shown in Fig. 9 differs from the flowchart of the recommendation system 100 included in the sequence diagram shown in Fig. 4 in that step S21 is included between step S11 and step S12.

[0079] In step S11, the task execution unit 110 receives a request G11 including text data from the user terminal 400. The task execution unit 110 supplies the received request G11 to the intention information generation unit 140.

[0080] In step S21, the intention information generation unit 140 causes the third language model 313 to generate intention information based on the request G11 received from the task execution unit 110. The intention information generation unit 140 supplies the intention information generated by the third language model 313 to the task execution unit 110.

[0081] In step S12, the task executing section 110 supplies the request G11 and the intention information to the first language model 311 to generate a first answer G21.

[0082] In step S13, the evaluation information acquisition unit 120 acquires evaluation information G200 related to each of the plurality of language models from the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit .

[0083] In step S14, the recommendation information generation unit 130 cooperates with the server 300 to cause the second language model 312 to generate recommendation information G22. The recommendation information generation unit 130 in this embodiment selects a recommended language model for generating a response to the request from multiple language models based on the request G11, the intention information, the first answer G21, and the evaluation information G200. Furthermore, the second language model 312 generates recommendation information G22 including a message presenting the recommended language model to the user.

[0084] Through the above-described processing, the recommendation system 100 in this embodiment generates a first answer G21 using the intention information generated from the request G11. This allows the recommendation system 100 to suppress a decrease in the generation quality of the first answer G21. Furthermore, the recommendation system 100 uses the intention information to generate recommendation information G22. This allows the recommendation system 100 to suppress a decrease in the generation quality of recommendation information.

[0085] Next, the flow of information processed by the recommendation system 100 according to the second embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing the flow of information in the recommendation system 100 according to the second embodiment.

[0086] When the task execution unit 110 receives the request G11 from the user terminal 400, it supplies the received request G11 to the intention information generation unit 140. The intention information generation unit 140 supplies the request G11 to the third language model 313, causing the third language model 313 to generate intention information G140. The intention information includes, for example, a request type, analysis data, and search words. The intention information generation unit 140 supplies the intention information G140 generated by the third language model 313 to the task execution unit 110.

[0087] The request type shown in FIG. 10 is "generate a summary." The request type of the intention information G140 has the same meaning as the request type of the evaluation information G200. The analysis data may include, for example, information about tokenized words, information about the language type (e.g., Japanese, English, etc.), and sentiment analysis of the text. The search words are, for example, words extracted from the request G11 and determined by the third language model 313 to require searching. The search words may also include search results.

[0088] The task execution unit 110 supplies the request G11 and the intention information G140 to the first language model 311, causing the first language model 311 to generate a first answer G21. Upon receiving the first answer G21 from the first language model 311, the task execution unit 110 supplies the received first answer G21, request G11, and intention information G140 to the recommendation information generation unit 130.

[0089] The recommendation information generation unit 130 receives the evaluation information G200 from the evaluation information acquisition unit 120. The recommendation information generation unit 130 also receives the request G11, the first answer G21, and the intention information G140 from the task execution unit 110. The recommendation information generation unit 130 supplies the received information to the second language model 312, causing it to generate recommendation information G22.

[0090] The second embodiment has been described above. With the above-described configuration, the recommendation system 100 generates a first answer G21 to a user request G11 while suppressing a decrease in generation quality. The recommendation system 100 also presents to the user a recommended language model, which is another language model that may have higher generation quality than the first language model 311. Therefore, according to this embodiment, it is possible to provide a recommendation system, a recommendation method, and a program that suitably propose an LLM that can appropriately respond to a user request.

[0091] <Third embodiment> Next, a third embodiment will be described. The third embodiment differs from the above-described recommendation system 100 in that the recommendation system 100 includes an evaluation unit 150.

[0092] 11 is a block diagram of a recommendation system 100 according to the third embodiment. The recommendation system 100 according to this embodiment mainly includes a task execution unit 110, an evaluation information acquisition unit 120, a recommendation information generation unit 130, an intention information generation unit 140, and an evaluation unit 150.

[0093] The evaluation unit 150 receives the request G11, the first answer G21, and the intention information G140 as input, and outputs an evaluation score for the first answer G21 generated by the first language model 311 to the fourth language model 314, which has an execution environment different from that of the first language model 311. The evaluation score is an index indicating the quality of the generated answer. As described above, the generation quality may include factors such as the accuracy, relevance, consistency, and customizability of the answer. In other words, the fourth language model 314 is configured to generate the evaluation score based on these factors.

[0094] The evaluation score is expressed by an integer ranging from 0 to 100, for example. In this case, for example, a higher value may be defined as a higher generation quality. However, the evaluation score is not limited to the above definition. The evaluation score may be quantitative or may be a qualitative indicator such as "good" or "bad."

[0095] The recommendation information generation unit 130 according to this embodiment generates the recommendation information G22 by taking the evaluation score into consideration. That is, the recommendation information generation unit 130 according to this embodiment generates the recommendation information G22 by adding at least the evaluation score generated by the evaluation unit 150 to the second language model 312. This allows the recommendation system 100 to cause the second language model 312 to generate the recommendation information G22 in a manner that allows easy comparison with the evaluation information G200 accumulated in the database 200.

[0096] The evaluation unit 150 according to this embodiment may have a function of supplying the request G11 and the evaluation score corresponding to the request G11 to a database for accumulation in the evaluation information. With this configuration, the response system 10 can enrich the evaluation information G200 by the user using the response system 10.

[0097] Next, the processing executed by the recommendation system 100 according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart of a recommendation method according to the third embodiment. The flowchart shown in Fig. 12 differs from the flowchart shown in Fig. 9 in that step S31 is included after step S12.

[0098] In step S12 according to this embodiment, the task executing unit 110 supplies the request G11 and the intention information G140 to the first language model 311 to generate a first answer G21. The task executing unit 110 supplies the request G11, the first answer G21 generated by the first language model 311, and the intention information G140 to the evaluation unit 150.

[0099] In step S31, the evaluation unit 150 supplies the received request G11, first response G21, and intention information G140 to the fourth language model 314, causing the fourth language model 314 to generate an evaluation score. The evaluation unit 150 supplies the evaluation score generated by the fourth language model 314 to the recommendation information generation unit 130.

[0100] In step S13, the evaluation information acquisition unit 120 acquires the evaluation information G200 from the database 200. The evaluation information acquisition unit 120 supplies the acquired evaluation information G200 to the recommendation information generation unit .

[0101] In step S14, the recommendation information generation unit 130 cooperates with the server 300 to cause the second language model 312 to generate recommendation information G22. The recommendation information generation unit 130 in this embodiment selects a recommended language model for generating an answer from multiple language models based on the request G11, the intention information G140, the first answer G21, the evaluation information G200, and the evaluation score generated by the evaluation unit 150. Furthermore, the second language model 312 generates recommendation information G22 including a message presenting the recommended language model to the user.

[0102] (Evaluation Information G200) Next, the evaluation information G200 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a second diagram showing the evaluation information G200. The evaluation information G200 shown in Fig. 13 differs from the evaluation information G200 shown in Fig. 5 in that it includes a score. The score is an evaluation of the performance response to the performance prompt.

[0103] For example, the evaluation information G200 includes the following information: The first information is that when user B sent a request "Please summarize sentence D" to language model "Model 1-ver. 3," he received an actual response "I will summarize it..." and the score of this actual response was 65. The second information is that when user B sent a request "Please summarize sentence D" to language model "Model 2-ver. 2," he received an actual response "Below is the summarized sentence..." and the score of this actual response was 39.

[0104] The third information includes content that when User C sent a request "Please generate code that executes..." to the language model "Model 1-ver. 3," the code generated by this language model was obtained as an actual answer. The third information also includes content that the score of this actual answer was 52.

[0105] The fourth information includes a content that when User C sent a request "Please generate code that executes..." to the language model "Model 1-ver. 2," the code generated by this language model was obtained as an actual answer. The third information includes a content that the score of this actual answer was 48.

[0106] By including the above information, the recommendation information generation unit 130 supplies the evaluation information G200 including the score to the second language model 312. This allows the second language model 312 to compare the evaluation score G150 corresponding to the first answer G21 with the evaluation information G200. Therefore, the recommendation information generation unit 130 can cause the second language model 312 to appropriately generate recommendation information including a recommended language model based on an objective evaluation.

[0107] Next, the flow of information processed by the recommendation system 100 according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a diagram showing the flow of information in the recommendation system 100 according to the third embodiment.

[0108] The task execution unit 110 according to this embodiment causes the first language model 311 to generate a first answer G21 using the intention information G140 received from the intention information generation unit 140. The task execution unit 110 supplies the request G11, the first answer G21, and the intention information G140 to the evaluation unit 150.

[0109] The evaluation unit 150 supplies the information received from the task execution unit 110 to the fourth language model 314. As a result, the evaluation unit 150 causes the fourth language model 314 to generate an evaluation score G150. In the example shown in FIG. 14, the evaluation score G150 indicates "Score: 65." In other words, the evaluation score G150 is 65. After causing the fourth language model 314 to generate the evaluation score, the evaluation unit 150 supplies the request G11, the first answer G21, the intention information G140, and the evaluation score G150 to the recommendation information generation unit 130.

[0110] The evaluation unit 150 also supplies additional information G151 to the database 200. The additional information G151 is supplied to the database 200 with the aim of being added to the evaluation information G200. That is, the additional information G151 is information corresponding to the evaluation information G200. The additional information G151 includes at least a request G11, a first answer G21, and an evaluation score G150 that correspond to each other. The additional information G151 may also include a portion of the intention information G140.

[0111] The recommendation information generation unit 130 according to this embodiment receives the request G11, the first answer G21, the intention information G140, and the evaluation score G150 from the evaluation unit 150, and also receives the evaluation information G200 from the evaluation information acquisition unit 120. The recommendation information generation unit 130 causes the fourth language model 314 to generate recommendation information G22 that takes into account the evaluation score G150.

[0112] The third embodiment has been described above. The recommendation system 100 according to this embodiment can select a more suitable recommended language model by using the evaluation score G150. Furthermore, the recommendation system 100 according to this embodiment can efficiently accumulate evaluation information G200. The recommendation system 100 can select a more suitable recommended language model by using the efficiently accumulated evaluation information G200. Therefore, according to this embodiment, it is possible to provide a recommendation system, a recommendation method, and a program that can suitably propose an LLM that can appropriately respond to a user request.

[0113] <Fourth embodiment> Next, a fourth embodiment will be described. In the fourth embodiment, the function of the recommendation information generation unit 130 is different from that of the above-described recommendation system 100. That is, the recommendation information generation unit 130 according to this embodiment outputs recommendation information including information about the difference between the first language model 311 and the recommended language model to the user terminal 400 used by the user who accepted the request G11.

[0114] Fig. 15 is a flowchart of a recommendation method according to the fourth embodiment. The flowchart shown in Fig. 15 differs from the flowchart shown in Fig. 12 in that it includes processes of steps S141 to S144 instead of the process of step S14.

[0115] In step S141, the recommendation information generator 130 causes the second language model 312 to select a recommended language model based on the evaluation score G150.

[0116] In step S142, the recommendation information generator 130 supplies the request G11 to the recommended language model selected by the second language model 312, causing it to generate a response to the request G11.

[0117] In step S143, the recommendation information generator 130 causes the second language model 312 to compare the answer of the recommended language model with the first answer G21 of the first language model 311.

[0118] In step S144, the recommendation information generator 130 causes the second language model 312 to generate recommendation information using the results of the above comparison.

[0119] An example of recommendation information generated by the recommendation information generation unit 130 according to the fourth embodiment will be described with reference to Fig. 16. Fig. 16 is a diagram showing recommendation information according to the fourth embodiment. The recommendation information generation unit 130 shown in Fig. 16 outputs first explanation information G31, second explanation information G32, and upsell information G23 as recommendation information.

[0120] The first explanation information G31 includes a message that compares the recommended language model with the first language model 311 and explains the advantages of the recommended language model. The recommendation information generation unit 130 causes the second language model 312 to generate the first explanation information G31 and outputs the generated first explanation information G31 to the user terminal 400. The first explanation information G31 shown in FIG. 16 includes a message that says, "Model 1-ver.3 is superior to Model 2-ver.2 in the following points: (1)..., (2)...."

[0121] By outputting the first explanation information G31, the recommendation system 100 can clearly indicate the advantages of the recommended language model to the user. Also, by outputting the first explanation information G31, the recommendation system 100 can preferably encourage the user to use the recommended language model.

[0122] The second explanation information G32 includes a message that compares the answer of the recommended language model with the first answer G21 of the first language model 311 and explains the advantages of the answer of the recommended language model. The recommendation information generation unit 130 causes the second language model 312 to generate the second explanation information G32 and outputs the generated second explanation information G32 to the user terminal 400. The second explanation information G32 shown in FIG. 16 includes a message that says, "Using model 1-ver.3 for this request will result in improvements in (3)... and (4)...."

[0123] By outputting the second explanation information G32, the recommendation system 100 can clearly indicate to the user specific improvements that can be made when using the recommended language model. Also, by outputting the first explanation information G31, the recommendation system 100 can appropriately encourage the user to use the recommended language model.

[0124] The recommendation information generation unit 130 can output upsell information G23 in addition to the above-described first explanation information G31 and second explanation information G32. By outputting recommendation information using such a combination, the recommendation system 100 can smoothly encourage the user to use the recommended language model.

[0125] The fourth embodiment has been described above. In steps S141 to S144 of the flowchart in Fig. 15, the recommendation information generation unit 130 may supply at least a part of the intention information G140 to the recommended language model. This allows the recommendation information generation unit 130 to input a prompt similar to that of the first language model 311 to the recommended language model.

[0126] As described above, according to the fourth embodiment, the recommendation system 100 can preferably encourage the user to use a language model with high generation quality.

[0127] (Answering System 10 Variation) Next, we will explain a modified example of the usage mode of the response system 10. Fig. 17 is a block diagram showing a variation of the response system 10. The response system 10 shown in Fig. 17 differs from the response system 10 shown in Fig. 1 in that the response system 10 shown in Fig. 1 is communicably connected to a business operator device 500 instead of a user terminal 400.

[0128] The business operator device 500 is a computer, server, cloud service, PaaS, SaaS, or the like for providing business-related services to multiple customers. The business operator device 500 is communicably connected to multiple customer terminals 600 via any communication means and provides the business operator's services. Here, the services provided by the business operator to customers include the answer service provided by the answer system 10. That is, the answer system 10 provides the answer service to the business operator's customers via the business operator device 500, as well as the functions of the recommendation system 100. With the above-mentioned configuration, the answer system 10 can provide the functions of the present disclosure to various businesses.

[0129] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0130] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate. [Explanation of symbols]

[0131] 10 Answer System 100 Recommendation System 110 Task Execution Unit 120 Evaluation information acquisition unit 130 Recommendation information generation unit 140 Intention information generation unit 150 Evaluation Department 200 databases 300 servers 311 First Language Model 312 Second Language Model 313 Third Language Model 314 Fourth Language Model 400 User Terminals 500 Operator equipment 600 customer terminals 1000 computers 1010 Bus 1020 processor 1030 memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface G11 Request G21 1st answer G22 Recommendation Information G23 Upsell Information G31 First explanatory information G32 Second explanatory information G140 Intent Information G150 evaluation score G151 Additional Information G200 Evaluation Information N1 Network

Claims

1. a task execution unit that receives a request including text data from a user; an evaluation information acquisition unit that acquires evaluation information regarding each of a plurality of language models from a predetermined database; a recommendation information generation unit that causes a language model to select a recommended language model for generating a response to the request from the plurality of language models based on the request and the evaluation information; an intention information generation unit that causes a language model to generate intention information to be input to a language model based on the request, for the purpose of supplementing the request; the task execution unit is a response system that inputs at least a part of the intention information into a language model and causes the recommended language model to generate a response to the request, an evaluation unit that receives the request, the response, and the intention information as input and outputs an evaluation score for the response generated by the language model to the language model; the recommendation information generation unit generates recommendation information including a message presenting the recommended language model to the user, taking the evaluation score into consideration; Answer system.

2. In the response system according to claim 1, the intention information generation unit generates the intention information including information regarding the type of the request from the context of the text data. Answer system.

3. In the response system according to claim 1, the intention information generation unit includes a search engine that searches for information within a predetermined network in cooperation with a language model, and causes the language model and the search engine to generate the intention information; Answer system.

4. In the response system according to any one of claims 1 to 3, the evaluation unit provides the request and the evaluation score corresponding to the request to the database for storage in the evaluation information; Answer system.

5. The computer Accept a request containing text data from a user, Obtaining evaluation information for each of a plurality of language models from a predetermined database; causing a language model to select a recommended language model for generating a response to the request from the plurality of language models based on the request and the evaluation information; generating intent information from a language model based on the request for input to the language model to supplement the request; a response method for receiving the request, the response, and the intention information as input, and causing a language model to output an evaluation score for the response generated by the language model, the method comprising: inputting at least a portion of the intention information into a language model to cause the recommended language model to generate a response to the request; generating recommendation information including a message presenting the recommended language model to the user in consideration of the evaluation score; How to answer.

6. Accept a request containing text data from a user, Obtaining evaluation information for each of a plurality of language models from a predetermined database; causing a language model to select a recommended language model for generating a response to the request from the plurality of language models based on the request and the evaluation information; generating intent information from a language model based on the request for input to the language model to supplement the request; a response method for receiving the request, the response, and the intention information as input, and causing a language model to output an evaluation score for the response generated by the language model, the method comprising: inputting at least a portion of the intention information into a language model to cause the recommended language model to generate a response to the request; generating recommendation information including a message presenting the recommended language model to the user in consideration of the evaluation score; The answering method is executed by a computer. program.

Images