Response output device

Abstract

To provide a more suitable response output technique.SOLUTION: A response output apparatus that outputs a response related to AI (AI) collectively transmits a prompt, which is the same question sentence created based on information input by a user, to a first group of a plurality of large-scale language models (LLMs) that are candidates for an answer source, and receives and acquires a plurality of answers including an answer of each LLM from the plurality of LLMs in the first group.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a technology for a response output device using artificial intelligence (AI). [Background technology]

[0002] A response output technology using artificial intelligence (AI) such as a language model is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special table 2019-528512 publication Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes that the model that inputs a question and outputs an answer is a model that is trained and generated based on a human dialogue corpus, and that the number of models is not limited to one, and may be multiple.

[0005] The disclosure of Patent Document 1 does not give sufficient consideration to a configuration for suitably providing a response output using artificial intelligence to a user.

[0006] The response output device sends a question prompt created based on user input to a server having a model such as a large-scale language model (LLM). The server generates and outputs an answer from the input question prompt using the model, and sends it to the response output device. The response output device receives the answer and outputs it to the user. In this case, if the response output device can use multiple models (LLMs) as the destination of the question, in other words, the source of the answer, there is room for consideration as to how to optimally use multiple models.

[0007] An object of the present disclosure is to provide a more suitable response output technology. [Means for solving the problem]

[0008] To solve the above problem, for example, the configuration described in the claims is adopted. The present application includes multiple means for solving the above problem, and one example thereof may be configured as follows: A response output device that outputs a response related to artificial intelligence (AI), which sends a prompt, which is the same question sentence created based on user input information, to a first group of multiple large-scale language models (LLMs) that are candidate answer sources in a batch, and receives / acquires multiple answers, including answers from each LLM, from the multiple LLMs in the first group. [Effects of the Invention]

[0009] According to the present invention, a more suitable response output technique can be provided. Other problems, configurations, and effects will become clear in the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows the configuration of a system including a response output device according to an embodiment. [Figure 2] 1 shows the configuration of multiple LLM servers and the like in a system including a response output device of an embodiment. [Figure 3] 1 shows an example of the configuration of a response output device according to an embodiment. [Figure 4] 10 shows the flow of a series of processes between a response output device and multiple LLM servers in a system including the response output device of the embodiment. [Figure 5A] 10 shows examples of setting items in the response output device of the embodiment. [Figure 5B] 10 shows a setting example in the response output device of the embodiment. [Figure 5C] 10 shows a setting example in the response output device of the embodiment. [Figure 5D] 10 shows an example of a screen display related to a user setting function in the response output device of the embodiment. [Figure 6] 10 shows an example of a condition setting process in the response output device of the embodiment. [Figure 7A] 10 shows an example of a screen display when a question is input in the response output device of the embodiment. [Figure 7B] 10 shows an example of a screen display when a question is input in the response output device of the embodiment. [Figure 8A] 10 shows an example of a screen display when a response is output in the response output device of the embodiment. [Figure 8B] 10 shows an example of a screen display when a response is output in the response output device of the embodiment. [Figure 9] 10 shows an example of an answer selection process in the response output device of the embodiment. [Figure 10A] An example of evaluation of two items in the response output device of the embodiment will be shown. [Figure 10B] 10 shows an example of evaluation of three items in the response output device of the embodiment. [Figure 10C] 10 shows an example of priorities based on evaluation results of two items in the response output device of the embodiment. [Figure 10D] 10 shows an example of priorities based on evaluation results of three items in the response output device of the embodiment. [Figure 11] 10 shows an example of a response display process in the response output device of the embodiment. [Figure 12A] 10 shows an example of primary detail display of an answer in the response output device of the embodiment. [Figure 12B] 10 shows an example of primary detail display of an answer in the response output device of the embodiment. [Figure 12C] 10 shows an example of primary detail display of an answer in the response output device of the embodiment. [Figure 12D] 10 shows an example of primary detail display of an answer in the response output device of the embodiment. [Figure 13A] 10 shows an example of secondary detailed display of an answer in the response output device of the embodiment. [Figure 13B] 10 shows an example of secondary detailed display of an answer in the response output device of the embodiment. [Figure 13C] 10 shows an example of secondary detailed display of an answer in the response output device of the embodiment. [Figure 13D] 10 shows an example of secondary detailed display of an answer in the response output device of the embodiment. [Figure 14] 10 shows an example of detailed display of an answer in the response output device of the embodiment. [Figure 15] 10 shows an example of detailed display of an answer in the response output device of the embodiment. [Figure 16] 10 shows an example of detailed display of an answer in the response output device of the embodiment. [Figure 17] 10 shows an example of response display control in the response output device of the embodiment. [Figure 18] 10 shows an example of detailed display of an answer in the response output device of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, identical parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, ranges, etc., in order to facilitate understanding of the invention.

[0012] For the sake of explanation, when describing processing by a program, the program, function, processing unit, etc. may be described as the main body, but the main hardware body for these is a processor, or a controller, device, computer, system, etc. that is configured with the processor, etc. A computer executes processing according to a program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. A processor is configured, for example, with semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing, but can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0013] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network or a non-transitory computer-readable storage medium, such as a memory card or disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information may be composed of structures such as, but not limited to, tables and lists. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0014] Note that if the artificial intelligence (AI) response output device according to this embodiment has a display screen / display function, it may be referred to as a display device. If the response output device has an audio output function, it may be referred to as an audio output device. The response output device may simply be referred to as an information processing device, a computer system, or the like. A system including the response output device and a server device such as an LLM server that stores an AI model such as a large-scale language model (LLM) may be referred to as an artificial intelligence (AI) response output system. Furthermore, if the response output device provides an AI response service to a user and assists the user, the response output device or the display output of the response output device can serve as an AI assistant for the user. Therefore, in this case, the response output device may be referred to as an AI assistant device or an AI assistant display device. Similarly, in this case, a system including the response output device and the LLM server may be referred to as an AI assistant system or an AI assistant display system. Furthermore, in this case, the response output device serves as an interface between the user and the AI, and therefore may be referred to as an AI interface device or the like. A system including this response output device and an LLM server may be called an AI interface system.

[0015] [Assignments, etc.] Provide additional explanation on issues, etc.

[0016] 1. The following can be considered as a system including a response output device of the comparative example. Assume that there are multiple learning models corresponding to multiple fields as multiple models available to the response output device. In other words, there are models (LLMs) that have been trained by specializing (fine-tuning, etc.) for each field. In other words, these models are specialized models, specialized models, field models, etc.

[0017] In the comparative example, the response output device switches the model to be used (i.e., the model to which the question is sent) so as to send a question prompt to a specialized model corresponding to an appropriate field according to the user's question. For example, if the content of a certain question is closely related to a certain field A, model A that has been trained to specialize in that field A is selected as the destination of the question. On the other hand, if the content of a certain question is not closely related to a specific field, a general-purpose learning model (in other words, a general-purpose model) is selected as the destination of the question.

[0018] Specialized model training for each field / area through fine-tuning and other methods can improve the accuracy of answers in that field / area, but it can also raise issues with credibility and hallucination due to bias in the training data. Hallucination is a phenomenon in which an LLM / generative AI generates false answers that are not based on facts. One approach to addressing credibility and hallucination is to use multiple models, such as using different LLMs for the destination (response source) of questions, as in the comparative example. However, such measures (using multiple models) can create issues with user operability and convenience, such as making user operations and decisions more complicated. There is a fundamental trade-off between measures to address credibility and hallucination and user operability.

[0019] In contrast to the comparative example, in the system of this embodiment, when multiple available models (e.g., multiple LLM servers) are available as candidate models, the response output device sends question prompts to the multiple models (LLM servers) in a batch. In other words, the response output device sends the same question in parallel to multiple LLM servers that are candidate answer sources. The candidate models (LLMs) may be specialized models trained for each field (e.g., field A model, field B model, ..., field Z model), or they may be general-purpose models or a mixture of both. The candidate models (LLMs), i.e., the LLM servers connected to the response output device via communication, may be LLM sites that require a fee (subscription contract) or free LLM sites.

[0020] In the system of this embodiment, the response output device receives and acquires answers from each of the multiple candidate LLMs. From these multiple answers, the response output device determines the priority of the answer (or the LLM that generated the answer) to be output to the user as the actual answer. The response output device selects an answer (sometimes referred to as an adopted answer, a preferred answer, etc.) to be output to the user as the actual answer in accordance with the priority. The response output device outputs the answer from the LLM that provided the answer, selected in accordance with the priority, to the user by display or audio.

[0021] As a result, in this embodiment, in addition to credibility and measures against hallucination, user operability and convenience are also realized. The user does not need to perform cumbersome operations or make decisions regarding multiple questions and multiple answers to multiple LLMs.

[0022] 2. In the system of this embodiment, the response output device has the functions of simultaneously sending question prompts to multiple candidate LLMs, receiving and acquiring multiple answers from the multiple LLMs, and selecting and creating an answer to be output to the user from the multiple answers based on priority, and outputting the answer to the user. The main features of this embodiment relate to the processing method of the response output device for simultaneously sending questions to multiple LLMs, the processing method for selecting an answer to be output from the multiple answers, and the processing method for outputting the selected answer to the user. Any known LLM can be used as the LLM to be used as the answer source, and the details are not important.

[0023] Prompts (in other words, questions or instructions) correspond to commands or instructions that users input to the AI ​​LLM, and specifically are sentences entered into the chat box / text field, or image information or audio information.

[0024] The LLMs used as candidates may be multimodal LLMs (multimodal AI), single-modal LLMs (single-modal AI), or a mixture of these. Multimodal AI is AI that can comprehensively process different types of data and information, such as text, images, and audio. Single-modal AI is AI that can process one type of data and information, such as text only, images only, or audio only.

[0025] 3. On the other hand, currently available LLMs based on general-purpose learning models (general-purpose LLMs) can generate answers to a variety of questions, but they have issues with the accuracy and precision of the answers. LLMs emphasize unlearned phenomena and contextual compatibility, which can lead to hallucination. Countermeasures against hallucination are important. On the other hand, when using specialized LLMs that have learned a specific field, the accuracy and precision of answers in the field studied can be improved, but conversely, they fall short of general-purpose models in terms of versatility.

[0026] There are practical limitations to increasing the accuracy of answers using a general-purpose model, due to resource limitations, etc. Therefore, one idea is to combine a general-purpose model with a specialized model for each field. If you want to achieve both versatility and accuracy, one idea is to switch the LLM (general-purpose LLM or specialized LLM) that answers the questions depending on the question, as in the comparative example.

[0027] In contrast, in this embodiment, instead of switching the LLM to which the question is sent as in the comparative example, the response output device first sends a question to multiple candidate LLMs at once and receives and acquires multiple answers (at least one answer per LLM) from the multiple LLMs. However, in this case, the response output device acquires a large number of answers, and outputting these answers directly to the user creates a problem. That is, when a user checks these answers, not only does the operation, work, and recognition become cumbersome and difficult, but the user also faces difficulty in determining which answers to trust and the reliability of each answer.

[0028] Therefore, this embodiment has the following solution etc.: First, for a prompt for a question created based on a user input, the response output device selects multiple LLMs to be the destination / response source of the question based on predetermined rules (destination selection rules) and conditions, and then transmits the prompt for the question to the selected candidate LLMs all at once.

[0029] Next, the response output device prioritizes the multiple answers to the question from the multiple LLMs based on predetermined rules (answer selection rules) and conditions, and selects the most appropriate answer that is estimated to be highly credible. From the multiple received answers, the response output device selects a smaller number, up to one or a few answers, up to a set number of output answers, as adopted answers / priority answers. The response output device outputs the selected answers to the user in order of priority.

[0030] For the sake of explanation, multiple LLMs that are candidates for sending questions and providing answers may be referred to as the first LLM, and multiple answers received from the first LLM may be referred to as the first answer (answer group, first group).Furthermore, the answer selected to be actually output in response to the first answer of the first LLM (adopted answer / priority answer) may be referred to as the second answer, and the LLM that generated the answer corresponding to the second answer may be referred to as the second LLM.

[0031] 4. Generally, LLM does not guarantee the reliability of answers. While increasing the reliability of answers, we also want to improve user usability, operability, and convenience. In order to increase the reliability of answers, we do not want to burden users with complicated operations. In the user interface provided by the response output device, we want to reduce the complexity of user operations and settings, as well as the complexity of the screen display. For example, we do not want to require users to perform operations or settings at deep levels in a multi-layered hierarchical structure.

[0032] In this embodiment, the response output device, particularly the control unit, that exists between the user (the person asking the question) and the multiple LLMs provides a user interface for the question-and-answer exchange of the AI ​​(LLM). The response output device provides, for example, a graphical user interface (GUI) screen display and voice input / output, and processes questions to the multiple LLMs and answers from the multiple LLMs through the user interface.

[0033] The user inputs question-related information (e.g., text, image, or voice input) into the user interface of the response output device, and the control unit creates a question prompt based on the input information. The control unit then transmits the question prompt to multiple candidate LLM servers in bulk and receives and acquires multiple answers from the multiple LLMs. The response output device then prioritizes the multiple answers based on predetermined rules and conditions and selects (in other words, narrows down) a small number of answers that are presumed to be highly credible, for example. The response output device then outputs the selected answers to the user via the user interface.

[0034] The control unit of the response output device automatically handles the tedious operations and settings that may be required to handle multiple LLMs. This eliminates the need for the user to perform tedious operations or settings. The user receives the answer output on the user interface. For example, on a GUI display screen, the user can view and confirm a small number of answers to a question selected according to priority. In this series of input and output processes, the user does not need to be aware of the existence of multiple candidate LLMs, nor does he need to check a large number of answers.

[0035] Devices that can be used as response output devices and user interfaces, or devices that can be connected to or linked to these, include mobile information terminals such as users' smartphones, wearable devices such as HMDs, PCs, home appliances, and dedicated devices installed in stores and public facilities, but are not limited to these.

[0036] Although a configuration is assumed in which a control unit (controller) built into the response output device processes the main functions (the functions of making batch calls and selecting answers), the configuration is not limited to this, and a control unit outside the response output device may process the main functions. For example, the system may be one in which a server or the like equivalent to the control unit is provided on a communication network outside the response output device, and the response output device operates in cooperation with the server or the like via communication.

[0037] In the system of this embodiment, initial settings such as minimum rules and conditions are required to use the functions provided by the response output device, but the user can review these settings at any time and can also apply default settings. The rules and conditions to be set include rules for selecting multiple destination LLMs (destination selection rules) and rules for selecting an answer to output from multiple answers (answer selection rules). These settings can also be changed depending on the user's usage characteristics, etc. These settings may also be automatically changed through learning, etc., depending on the user's usage characteristics, etc.

[0038] Destination selection rules are conditions such as which LLM site to use as the destination for questions, and can also set the number of LLM sites to reference and the number of answers to reference per site. Specific LLM sites can also be individually specified and turned on / off. Answer selection rules, for example, can set the conditions (perspective, algorithm, etc.) to use to prioritize answers, and can also set the number of answers to output.

[0039] The response output device may automatically select and output an answer, and in so doing, may include, in the user interface, information (detailed information) such as the reason for selecting the answer and its evaluation (priority or evaluation value based on rules) along with the selected and output answer. The user may be able to receive the detailed information in response to a predetermined operation in the user interface. The response output device may also output and receive, in response to a predetermined operation by the user, answers and detailed information that were not selected and output from among the received answers.

[0040] The response output device may display information (detailed information) on the user interface, such as the number and names of LLM sites to which the question was sent, the number and names of LLM sites from which the answer was obtained, and the score (evaluation value) and priority of the answer, depending on the user's settings and operations.

[0041] The response output device selects answers to output from the multiple answers received or acquired according to a set number (referred to as the number of output answers, etc.). The response output device selects answers to output based on priority so that the number of output answers is equal to or less than the number of output answers. This number of output answers can be variably set according to the user. A user can adjust the amount of answer information they can receive by setting the number of output answers according to their time availability, cognitive ability, etc. For example, a user who has time can set the number of output answers to a larger number (e.g., several answers), thereby allowing them to view multiple answers in parallel / sequentially in order of priority. On the other hand, a user who does not have time can set the number of output answers to a smaller number (e.g., one answer), thereby allowing them to view only the single answer with the highest priority. The user can change the setting of this number of output answers as appropriate. Note that the number of output answers is the number of answers, but is not limited to this and may be determined by a reduction rate, etc.

[0042] The response output device may also allow the user to receive information about answers (rejected answers) that were not selected as answers to be output (accepted answers) according to predetermined rules among multiple answers received and acquired from multiple LLMs. The response output device retains information (part of the detailed information) about rejected answers and outputs the information about the rejected answers in response to a request or operation from the user. If the user wishes to view detailed information about the rejected answers, the user can view the detailed information by performing additional operations. The response output device normally automatically provides a small amount of information (i.e., only the selected answer) without requiring additional operations, and allows the user to view all detailed information (in other words, related information) including rejected answers and interaction history if desired, even if additional operations are required.

[0043] 5. The response output device connects via communication with multiple candidate LLMs (LLM servers / LLM sites). The control unit of the response output device sends prompts for the questions entered or specified by the user to the selected multiple LLMs in a batch. Note that this batch sending does not need to be strictly simultaneous, as long as it is roughly simultaneous. In other words, request data (for example, data in any format, such as packets on a communication network) corresponding to the question prompts can be processed sequentially so as to be sent to each LLM within a short period of time. For example, in a packet of request data containing a prompt, the address of the selected LLM server is written as the destination, and the address of the response output device is written as the source.

[0044] Furthermore, even if request data is sent from the response output device to multiple LLMs simultaneously, the time it takes to receive the response data from each LLM may differ. This is due to differences in the performance and load of each LLM server, differences in communication conditions, etc. The time at which the response output device receives multiple responses from multiple LLMs does not necessarily have to be simultaneous, and there may be a waiting time interval between receiving multiple responses. The response output device may determine priorities, etc. in real time according to the reception and acquisition of responses from the LLMs, and display / update the display of prioritized responses.

[0045] In addition, there may be cases where the series of processes from the input of a question to the output of an answer by the control unit of the response output device takes a relatively long time (for example, longer than the expected standby time). In such cases, for example, if a response from a certain LLM is slow and has not yet been obtained, the response output device may output information to inform the user of the situation and an estimated completion date (such as the estimated time required for the response to be obtained / output). This is effective because it allows the user to easily understand the situation. For example, the response output device may display in the user interface that a response from a certain LLM has not yet been obtained. The response output device may also display in the user interface the estimated time required for the response to be obtained / output from a certain LLM. The response output device may also predict, for each LLM, the time required from the issuance of a question to the reception of an answer, based on past experience and history information.

[0046] Furthermore, even when the response output device has not yet received or acquired all of the multiple responses from the multiple candidate LLMs, it may select an answer to output from one or more previously acquired answers in accordance with a predetermined rule and output the selected answer to the user. That is, the answer to output is selected in chronological order as new answers are received or acquired, and the answer to be output is updated sequentially. When a sufficient amount of time corresponding to the standby time has passed, multiple answers will be received or acquired from the multiple candidate LLMs, and an answer selected from them will be output.

[0047] The basic rule is to refer to one answer per LLM for a question (referred to as the number of reference answers), but as a variant, multiple answers may be referenced per LLM for a question. In other words, it may be possible to receive and obtain multiple answers from one LLM. If the number of reference answers is one, multiple answers can be obtained, the same number as the number of LLM sites from which the answers originate. If the number of reference answers is multiple, a larger number of answers can be obtained by multiplying the number of LLM sites from which the answers originate by the number of reference answers. In either case, the number of answers received and obtained is reduced by the set number of output answers, and the answer to be output is selected.

[0048] 6. The control unit of the response output device compares the content of the responses from multiple LLMs and prioritizes the responses based on predetermined rules and conditions. Based on the priorities, the control unit selects answers (or summary answers) from the multiple answers up to a predetermined number of output answers, and outputs the selected answers (adopted answers) on the user interface.

[0049] The control unit of the response output device may extract and compare the main parts (particularly the gist) of the content of each answer when making the comparison. When extracting the gist, an AI with a function for extracting gist may be used.

[0050] Furthermore, when making the comparison, the response output device may create a summary for each response and compare the summaries. When creating a summary, an AI (LLM) with a function for creating a summary may be used. In addition to the LLM (LLM server) used as the response source, there may be an LLM or AI used for a predetermined function such as summarization. The response output device may use not only an LLM connected to an external network for communication, but also an LLM (local LLM) or AI (e.g., priority processing AI 170 in FIG. 3) implemented within the response output device as the response source LLM or an LLM with a predetermined function.

[0051] Furthermore, the response output device may output the gist and summary information of each answer to the user in addition to outputting the answer itself.

[0052] 7. Methods and elements for prioritizing answers from multiple answers based on predetermined rules (answer selection rules) and conditions include, for example, the following:

[0053] (a) The fee structure of the respondent's LLM server / LLM site (paid / free, etc.). For example, prioritizing information from paid sites over information from free sites.

[0054] (b) Differences in the type of LLM the respondent is considering, such as general LLM / specialized LLM. For example, prioritizing information on specialized LLMs over general LLMs.

[0055] (c) Credibility: User evaluation information about the LLM server / LLM site from which the answer originates. For example, refer to word-of-mouth evaluation information (website information) from general users about the LLM site. Alternatively, refer to evaluation information about the LLM site by the user (yourself) of the response output device. Based on this evaluation information (one or both), priority is given to information from LLM sites with high evaluation points.

[0056] The user of the response output device can evaluate the outputted answer and input evaluation information (e.g., evaluation points) for the answer / LLM that generated the answer in the user interface. The response output device retains such evaluation information and calculates the evaluation value of the answer / LLM site. When selecting an answer based on the answer selection rules and conditions, the response output device reflects the evaluation value and prioritizes answers. Trustworthiness, as described below.

[0057] (d) Majority vote: Taking into account the differences in the answers from multiple LLMs, a majority vote will be used to determine the preferred answer.

[0058] (e) Validity: Give priority to answers that have clear grounds and evidence. In other words, give priority to answers that have high validity of reasoning in the LLM.

[0059] For example, the priority may be determined using one method or condition selected from (a) to (e) above, or multiple methods and conditions may be combined to determine the priority. When multiple methods and conditions are combined, the priority may be determined by, for example, weighting the evaluation value of each method or condition and calculating an overall evaluation value (total score).

[0060] For the methods and conditions such as those in (a) to (e) above, the control unit of the response output device may successively learn based on historical information of answers, etc., and update the contents of the methods and conditions. For example, when using the credibility of (c), the response output device learns the evaluation information of each LLM / answer, and as a result of the learning, the credibility of the LLM is updated.

[0061] <Example> A more detailed embodiment will be described below with reference to FIG. 1 and subsequent figures.

[0062] [System including response output device] FIG. 1 shows a system including a response output device 1 of this embodiment. The response output device 1 is an AI response output device / LLM response output device. The system of this embodiment is a system in which the response output device 1 used by user U1 communicates with an external LLM server 2 or the like. The response output device 1 is appropriately connected to multiple LLM servers 2 {2A, 2B, 2C, ..., 2Z} via a communication network 9. Each LLM server 2 (in other words, an LLM site) has an LLM. The communication network 9 is the Internet or the like, but the details are not limited thereto.

[0063] The response output device 1 includes a control unit (in other words, a controller) 11, a processing unit 12, a display unit (display) 13, an audio unit 14, and an operation input unit 15. The display unit 13, the audio unit 14, and the operation input unit 15 form a user interface 16.

[0064] The control unit 11 is configured to include a processor, a memory, etc. The control unit 11 controls the entire response output device 1 and each unit. The processor executes processing in accordance with a program read into the memory. This realizes the processing unit 12. The processing unit 12 includes processes such as a prompt transmission process 51 and a received response selection process 51.

[0065] In the example of FIG. 1, the response output device 1 has a display unit 13. The display unit 13 may be a flat display such as a liquid crystal display, a screen (projector) that projects an image from the rear, or a virtual image display device such as a head-up display. The display unit 13 may be a floating image display device that forms an optical image as a real image in the air. The display unit 13 may be a plasma display or an organic EL display in which pixels emit light themselves. The display unit 13 may be a touch panel equipped with a touch sensor.

[0066] The display unit 13 has a user input display 53 and an AI output display 55. The user input display 53 (in other words, a prompt display or question display) is a chat box or the like, and displays a question prompt based on the input of the user U1. The user input display 53 includes a display of text / images 54 constituting the question. The user input display 53 displays, for example, an icon representing the user U1, text and images (which may be videos) such as natural language or software code as components of the prompt, etc. The AI ​​output display 55 (in other words, an answer display) displays an answer from the AI ​​(LLM), etc. The AI ​​output display 55 includes a display of text / images 56 constituting the answer. The AI ​​output display 55 displays an icon representing the AI ​​(LLM), text and images (which may be videos) such as natural language or software code as components of the response from the AI ​​(LLM), etc. Note that this display configuration of the display unit 13 is merely an example and is not limited to this.

[0067] The audio unit 14 is a part that inputs the voice of the user U1 and outputs the voice from the AI ​​(LLM) and the response output device 1. In the case of an implementation that supports the voice input / output function, the audio unit 14 is provided. The response output device 1 includes an audio input unit 104 including a microphone, an audio output unit 105 including a speaker, etc., as shown in FIG. 3 described later. The audio unit 14 is realized by the audio input unit 104, the audio output unit 105, etc. The response output device 1 inputs the voice uttered by the user U1 through the audio input unit 104 and converts it into text by speech recognition. The response output device 1 also synthesizes the text of the answer from the AI ​​(LLM) and outputs it from the audio output unit 105.

[0068] The operation input unit 15 is a part that includes or is externally connected to an input device (e.g., a keyboard) for the user U1 to input text, etc. For inputting text / images, the display unit 13 may be used as a touch panel to enable touch operation input.

[0069] The response output device 1 can acquire user input information that is the basis for the prompt to be sent to the LLM through input via the display unit 13, audio unit 14, operation input unit 15, etc. The control unit 11 (particularly the prompt sending process 51) creates a prompt for the question based on the user input information and sends the prompt to the LLM server 2 that is the source of the answer.

[0070] The response output device 1 includes a communication unit 17 shown in Fig. 3, which will be described later. The communication unit 17 is a part that implements a communication interface for communicating with an external device through the communication network 9. The communication unit 17 includes, for example, a wireless antenna 17b. In the example of Fig. 1, the response output device 1 is wirelessly connected to a communication device 9b of the communication network 9 through the wireless antenna 17b of the communication unit 17.

[0071] The response output device 1 can communicate with a communication device 9b connected to the communication network 9, an LLM server 2 located beyond the communication device 9b, other servers 3, etc., via the communication unit 17. In the example of FIG. 1, the communication with the communication device 9b is shown to be wireless, but wired communication is also possible. The communication path between the response output device 1 and the LLM server 2 may include wired and wireless portions, or may go via a communication device such as a router or repeater. An example of another server 3 is a website that lists word-of-mouth reviews from general users about the LLM (LLM server 2).

[0072] The response output device 1 is appropriately connected for communication with the LLM server 2 via a communication network 9 or the like. The response output device 1 transmits data (request data) of a question prompt 201 to the destination (response source) LLM server 2. The response output device 1 receives data (response data) of an answer 202 from the LLM server 2.

[0073] For example, under the control of the control unit 11 of the response output device 1, a signal corresponding to the request data of the prompt 201 is transmitted to the outside from the wireless antenna 17b of the communication unit 17. The request data 201 corresponding to the signal is received via a communication device 9b (e.g., a wireless access point) 9b of the communication network 9 and transmitted to the destination LLM server 2 via communication such as IP of the communication network 9. The destination LLM server 2 receives the request data 201. The LLM provided in the LLM server 2 inputs the prompt 201, processes it, and generates a response. The LLM server 2 transmits the generated response 202 as response data to the response output device 1, which is the sender, via the communication network 9. The response output device 1 receives and acquires the response data of the response 202 via the communication unit 17.

[0074] Note that a configuration including the response output device 1 and the LLM server 2 as components as shown in FIG. 1 may be regarded as one system (for example, an AI response output system, an AI conversation system, etc.).

[0075] [LLM] A supplementary explanation will be given on large language models (LLMs). Specifically, various LLM models have been published, including GPT-1, GPT-2, GPT-3, InstructGPT, and ChatGPT. These technologies can also be used in this embodiment. These LLMs are artificial intelligence models generated by large-scale pre-training on the natural language contained in numerous documents and texts in the human world. The number of parameters in these artificial intelligence models exceeds 100 million. In addition to this, there are also models that have undergone reinforcement learning based on human feedback. An example of a base model is a model called Transformer. Reference 1, for example, has been published as an example of the learning of these models.

[0076] [Reference 1] Long Ouyang, et. al. “Training language models to follow instructions with human feedback”, https: / / arxiv.org / pdf / 2203.02155.pdf

[0077] These LLMs are capable of natural language translation, proofreading, and text summarization. Advanced LLMs are capable of natural language question answering (also known as dialogue or conversation), proposal generation, and programming code generation. Because these AI models have a very large number of parameters, training requires vast amounts of data and computational resources. Therefore, training this level of AI for a specific application (such as a field) is extremely resource-inefficient. Therefore, large-scale pre-trained LLMs have been created as foundation models that can be applied to a variety of applications. Duplicating such foundation model LLMs and using them on individual servers, etc., can be resource-efficient. Even foundation models can be configured to perform additional training, such as transfer learning, on individual servers, etc., depending on the application and purpose.

[0078] LLMs can pre-train natural language (text information) and perform input / output processing for natural language. Furthermore, multimodal LLMs, which can process information other than natural language (text information), such as images and audio, can also be applied. Examples of multimodal LLMs include GPT-4 (Reference 2) and Gato (Reference 3). These multimodal LLMs are artificial intelligence models generated through large-scale pre-training on images and audio, in addition to the numerous textual information in natural languages ​​present in the human world. Furthermore, some models employ reinforcement learning based on human feedback. Information other than natural language text information, such as images and audio, may also be referred to as non-natural language information sources.

[0079] [Reference 2] Open AI “GPT-4 Technical Report”, https: / / cdn.openai.com / papers / gpt-4.pdf [Reference 3] Scott Reed, et. al. “A Generalist Agent”, https: / / arxiv.org / pdf / 2205.06175.pdf

[0080] For the purposes of explanation, unless otherwise specified, "large-scale language model (LLM)" is a general term that includes various LLMs, without distinguishing between external LLMs and local LLMs, between general-purpose and specialized models, or between single and multimodal models.

[0081] The LLM server 2 in Fig. 1 includes an LLM capable of answering questions as described above. The LLM may be configured to be available on various terminals via an API (Application Programming Interface). The response output device 1 in Fig. 1 uses the LLM of the LLM server 2 via the API.

[0082] [Multiple LLMs] FIG. 2 is an explanatory diagram of multiple LLMs. FIG. 2 shows an example configuration of multiple LLM servers (LLM sites) 2 in FIG. 1. In the example of FIG. 2, multiple LLM servers 2 connected to a communication network 9 include LLM server 2A (LLM site A), LLM server 2B (LLM site B), LLM server 2C (LLM site C), etc. The number of LLM servers 2 may vary. Each LLM server 2 has a different LLM. For example, LLM server 2A has LLM-A, LLM server 2B has LLM-B, and LLM server 2C has LLM-C.

[0083] Each LLM server 2 performs processing including a receiving prompt process 2001, an LLM process 2002, and an answer transmission process 2003. The LLM server 2 performs the receiving prompt process 2001 to receive a prompt 201 from the response output device 1, performs the LLM process 2002 to input the prompt 201 into the LLM to generate an answer, and performs the answer transmission process 2003 to transmit the generated answer 202 to the response output device 1. The "common prompt" shown in the figure corresponds to the question prompt 201.

[0084] In the example of Figure 2, a prompt (request data) 201 for the same question is sent from the response output device 1 to each of the LLM servers 2 (2A, 2B, 2C) at LLM sites A, B, and C. In response to the prompt 201, answers (response data) 202 are generated: answer A from LLM site A, answer B from LLM site B, and answer C from LLM site C. The response output device 1 receives and acquires these answers (response data) 202. The response output device 1 selects an answer to output from the multiple answers it has received and acquired, and outputs the selected answer to user U1 on a user interface 16, such as a display unit 13 and an audio unit 14.

[0085] 1 and 2, the response output device 1 itself does not have an LLM, and uses an LLM server 2 as an external LLM. However, the response output device 1 may have an LLM (local LLM) and use the local LLM provided in the response output device 1 as part of multiple LLMs. The response output device 1 may use both the local LLM and the external LLM as candidates.

[0086] Furthermore, the LLM of each LLM server 2 is not limited to a single-modal LLM, and may be a multi-modal LLM.

[0087] [Example of response output device configuration] 3 shows an example configuration of the response output device 1. The response output device 1 includes a control unit 11, a memory 101, a nonvolatile memory 102, a storage 103, a display unit 13, a communication unit 17, an audio input unit 104, an audio output unit 105, an audio signal input unit 106, a video signal input unit 107, an imaging unit 108, an operation input interface 111, a power supply 112, a secondary battery 113, an audio control unit 114, a video control unit 115, etc., which are interconnected by an architecture such as a bus. The response output device 1 may also include a priority processing AI 170, a local LLM processing unit 180, etc.

[0088] The processor of the control unit 11 loads programs and the like read from the nonvolatile memory 102 or storage 103 into the memory 101 and executes program processing. The nonvolatile memory 102 stores setting information and the like. The storage 103 stores data such as text, video, and audio.

[0089] The audio input unit 104 includes a microphone, an audio processing circuit, etc., and picks up and inputs the audio of the user U1 using the microphone. The audio output unit 105 includes a speaker, an audio processing circuit, etc., and outputs audio such as responses and user interface information from the speaker. The audio signal input unit 106 can input audio data from an external audio source. The video signal input unit 107 can input video data from an external video source. The imaging unit 108 includes a camera, etc. The operation input interface 111 connects to an input device such as a keyboard. The power supply 112 stores power in a secondary battery 113 based on input from an external power source, and supplies the power from the secondary battery 113 to each unit.

[0090] The voice control unit 114 performs control processing related to voice input and voice output. For example, the voice control unit 114 converts voice input from the voice input unit 104 into text by voice recognition. For example, the voice control unit 114 converts text of a response from the LLM into voice by voice synthesis. The video control unit 115 performs control processing for displaying images and videos on the screen of the display unit 13 (display). For example, the video control unit 115 displays question prompts and responses from the LLM on the screen.

[0091] [Priority Processing AI] In this embodiment, the response output device 1 includes a priority processing AI 170. As will be described in detail later, the priority processing AI 170 is a part that performs the evaluation and calculation as AI processing when the above-mentioned majority vote or validity evaluation and calculation is applied during the process of determining the priority of answers and selecting an answer (received answer selection process 52).

[0092] The priority processing AI 170 may be integrated into the control unit 11. Alternatively, the priority processing AI 170 may be implemented as a type of local LLM. In a modified example, the priority processing AI 170 may be provided in a server or the like external to the response output device 1. When an external priority processing AI 170 is used, the response output device 1 accesses the server of the external priority processing AI 170 when necessary, transmits a processing request, and obtains a response of the processing result from the server.

[0093] The priority processing AI 170 performs majority voting or validity evaluation and calculation processing as priority processing using AI for multiple answers received and acquired from multiple LLM servers 2. When applying the majority voting method and conditions, the priority processing AI 170 calculates an evaluation value (score) for the content of each answer based on the majority voting algorithm. When applying the validity method and conditions, the priority processing AI 170 calculates an evaluation value (score) for the content of each answer based on the validity algorithm.

[0094] [Local LLM] In a modified example, a local LLM processing unit 180 may be provided inside the response output device 1. The LLM (local LLM) provided in the local LLM processing unit 180 can be used as one of the candidates for the source (destination) of the answer to the question. There may be multiple local LLMs inside the response output device 1. When the local LLM of the local LLM processing unit 180 is used as a candidate, the control unit 11 transfers the question prompt to the local LLM processing unit 180, and the local LLM processing unit 180 inputs the prompt to generate an answer and transfers the generated answer to the control unit 11.

[0095] [Basic flow] 4 shows the basic flow of a series of processes between a response output device 1 and multiple LLM servers 2 (e.g., LLM sites A, B, and C), from question input to answer output, in other words, a sequence diagram. In FIG. 4, an example is shown in which three LLM sites, A, B, and C, are used as candidates for the question destination and answer source. Using FIG. 4, a characteristic function of this embodiment will be described, namely, the function of sending a question to multiple LLMs at once, selecting an answer from the multiple received answers by prioritizing them based on predetermined rules and conditions, and outputting the selected answer to the user.

[0096] In step S1, after startup, the response output device 1 sets conditions related to the functions of this embodiment using the control unit 11. The conditions here are shown in FIG. 5A and other figures, which will be described later, and are setting information corresponding to predetermined rules, conditions, etc. The setting of the conditions may be an automatic setting (such as a default setting) by the control unit 11, or a user setting by the user U1. In step S2, the control unit 11 updates the conditions (setting information) as necessary. The update of the conditions may be an automatic update by the control unit 11, or a user setting by the user U1.

[0097] In step S3, the control unit 11 creates a prompt for the question based on the user-input information. In step S4, the control unit 11 selects and determines multiple LLM servers 2 (e.g., LLM sites A, B, and C) to be destinations (in other words, candidate answer sources) for the created prompt based on predetermined rules (destination selection rules) and conditions. The rules may specify multiple LLMs to be destinations in advance. The control unit 11 transmits the prompt (in other words, request data 201 including the prompt) to the multiple candidate LLM servers 2 in a batch via the communication unit 17.

[0098] Furthermore, in steps S3 and S4, the response output device 1 may display information about a plurality of LLMs as destination information on the display screen (prompt generation screen) of the display unit 13. The user can also view this destination information as needed, allowing the user to confirm and recognize a plurality of LLMs that are candidates for the response source.

[0099] In step S4, the response output device 1 transmits prompts 201 corresponding to the same question to multiple candidate LLM servers 2 in a batch. In detail, the response output device 1 transmits multiple pieces of request data 201 containing the same prompt to multiple LLM servers 2 with different destinations within a sufficiently short period of time. These pieces of request data are transmitted approximately simultaneously. This is not a limitation, and the request data may be transmitted sequentially, for example, with a slight time lag. In any case, the user U1 does not need to be aware of such background processing when transmitting in a batch, and can realize the batch transmission with a simple operation and a single operation.

[0100] Each of the multiple LLM servers 2 (e.g., LLM sites A, B, and C) that are the destinations of the request receive the prompt (request data) 201. In step S5, each LLM server 2 performs a process to receive the prompt 201 (receive prompt process 2001 in FIG. 2). In step S6, each LLM server 2 performs a process to input the prompt 201 into the LLM and generate a response (LLM process 2002 in FIG. 2). In step S8, each LLM server 2 performs a process to transmit the generated response (in other words, response data 202 based on the response) to the response output device 1 that originated the request (transmit response process 2003 in FIG. 2).

[0101] The timing and time required for processing may differ for each LLM server 2. This will be described later.

[0102] In step S8, the control unit 11 of the response output device 1 performs processing to receive and acquire responses (response data) 202 from each of the multiple destination LLM servers 2 via the communication unit 17. These multiple responses may be referred to as a first group, a group of responses, etc. In step S8, the control unit 11 may set a predetermined standby time to receive and acquire multiple responses.

[0103] In step S9, the control unit 11 prioritizes the answers from the multiple answers (answer group) received and acquired in step S8 based on predetermined rules (answer selection rules) and conditions corresponding to the conditions (setting information) set in steps S1 and S2, and selects an answer to be output (prioritized answer / adopted answer). At this time, the control unit 11 determines the priority / priority among the answers based on predetermined rules and conditions (e.g., evaluation methods such as credibility, majority vote, or validity) described below. Then, the control unit 11 selects an answer to be output from the multiple answers according to the priority, within a range up to the set number of output answers. The number of output answers (in other words, the number of displayed answers) is, for example, from one to several.

[0104] In step S10, the control unit 11 outputs the answer (priority answer / adopted answer) selected in step S9 to the user U1 on the user interface 16 such as the display unit 13. At this time, information about the LLM that provided the priority answer may also be output at the request of the user U1.

[0105] This is the end of the processing flow for one question. If the next question by user U1 continues, the process returns to step S2 and is repeated in the same manner.

[0106] [Condition Settings] Figure 5A shows an example of the configuration for the condition setting (setting information) in step S1. The table in Figure 5A shows a list of condition setting items. The column items in the table include number, category, setting item, initial value / setting value, credibility, remark 1, and remark 2. The category is "sending" or "receiving." The "sending" item (row) is related to step S4 etc. and corresponds to the destination selection rule (in other words, the destination selection condition). The "receiving" item (row) is related to step S9 etc. and corresponds to the answer selection rule (in other words, the answer selection condition). The setting items include the number of LLM sites to use, the designation of the LLM site to use, the number of displayed answers (= the number of output answers), the number of reference answers per site, the target LLM site, priority conditions, etc.

[0107] The "Number of LLM sites to use" item is an item for setting the number of LLM sites (LLM servers 2) to be used as candidate response sources, in other words, the number of LLM sites to which prompts will be sent in bulk. When the maximum number of available LLM sites that can be connected to the response output device 1 for communication is N, the "Number of LLM sites to use" can be set to a value within the range of 1 to N. In this example, the setting value for the "Number of LLM sites to use" is shown as 3.

[0108] The "Specify LLM Sites to Use" item allows you to individually specify and set LLM sites to be used as candidate respondent sources based on the maximum number (N) and the "Number of LLM Sites to Use" (3). In this example, the three LLM sites corresponding to the "Number of LLM Sites to Use" = 3 are designated and set as LLM sites A, B, and C in Figure 2 from sites 1 to N based on their "trustworthiness" (user ratings). In other words, an example of a destination selection rule is to select multiple LLM sites corresponding to the "Number of LLM Sites to Use" as destinations in descending order of "trustworthiness." Alternatively, the destination selection rule may specify and set LLM sites to use based on some other perspective, or arbitrarily / freely, without considering "trustworthiness." A fixed number of LLM sites may also be set.

[0109] Furthermore, as an additional setting, the destination may be specified and set from another LLM site (for example, an LLM site with a lower level of trust other than LLM sites A, B, and C). In this case, the response output device 1 displays, for example, on the user interface 16, a list of connectable LLM sites corresponding to the number N, and user U1 can specify and set the destination from that list. Depending on the additional setting, the number of available LLM sites may increase, for example, from 3 to a larger number.

[0110] The "Credit Rating" column shows, for example, scores, which are evaluation values, as examples of user evaluation information for each LLM site. Each score is in the range of 0 to 100 points. User evaluation information is word-of-mouth evaluation information from general users, or evaluation information from user U1 (oneself) of this response output device 1. Based on this evaluation information, the credit rating is a score calculated so as to be normalized to the range of 0 to 100 points. For example, the default value (intermediate value) of the credit rating score is 50 points. For example, LLM site A (LLM-A) is 90 points, LLM site B (LLM-B) is 50 points, and LLM site C (LLM-C) is 30 points.

[0111] As noted in the notes, for example, LLM Site A (LLM-A) is a paid site (e.g., subscription) and a general-purpose LLM. LLM Site B (LLM-B) is a paid site (e.g., individual, pay-as-you-go) and a specialized LLM. LLM Site C (LLM-C) is a free site and a general-purpose LLM.

[0112] In "Receive," the "Number of Answers to Display" item sets the number of answers (priority answers) to be displayed in the user interface 16, and is set to a number smaller than the "Number of LLM Sites Used" (e.g., 3). The "Number of Answers to Display" can be set within the range of 1 to M, where M is the "Number of LLM Sites Used." The "Number of Answers to Display" can be set to the same value as M (e.g., 3), but if you want to enhance the effectiveness of the function of this embodiment, set the "Number of Answers to Display" to a value smaller than M (e.g., 2 or 1). In this example, the "Number of Answers to Display" is set to 1.

[0113] The "Number of reference answers per site" item sets the number of answers to be referenced as candidates for each LLM site (LLM server 2). This "number of reference answers" can be set within the range of 1 to L, where L is the maximum number of answers that one LLM site can generate at one time. In this example, the "number of reference answers" uses the default setting value of 1. In this case, one answer is referenced for each LLM site. In a modified example, the number of reference answers may be different for each candidate (destination) LLM site.

[0114] In this example setting, "Number of LLM sites used" = 3, "Number of displayed answers" = 1, and "Number of reference answers" = 1. Therefore, three answers (there is a one-to-one correspondence between LLM sites and answers) are obtained from three candidate LLM sites A, B, and C, and one answer will be selected and displayed from these based on priority.

[0115] The "Target LLM Sites" in "Receive" are the same as the multiple LLM sites (e.g., LLM sites A, B, and C) specified in "Specify LLM Sites to Use" in "Send." Responses from these "Target LLM Sites" are subject to evaluation to assign priorities, and the evaluation method and conditions are specified and set in "Priority Conditions."

[0116] The "Priority Condition" item is the method or condition (in other words, algorithm, evaluation item, etc.) for evaluating or determining the priority of answers corresponding to the answer selection rules or conditions. In this example, the "priority conditions" are (1) credibility, (2) majority vote, and (3) validity of inference. In this setting example, a combination of these three conditions is applied. Each condition is set to ON (applied). An evaluation value (score) is calculated for each condition, and priorities are assigned based on the overall evaluation value (score).

[0117] (1) The credibility score is a score (rating value) based on the credibility of the target LLM site (e.g., user rating information). This credibility score is the same as the score based on user rating information shown in the "Credibility" column, but it may also be calculated separately based on that score. For example, this credibility condition may be such that paid sites are given higher scores than free sites, or specialized LLMs are given higher scores than general-purpose LLMs.

[0118] In this embodiment, the trust condition is a case where the trust is calculated by reflecting both the word-of-mouth evaluation information of general users (other people) on external sites (other servers 3) and the evaluation information of user U1 (oneself) on the response output device 1. This is not limiting, and the trust may be calculated using only the word-of-mouth evaluation information of general users, or only the evaluation information of user U1. In a modified example, the word-of-mouth evaluation information of general users and the evaluation information of user U1 may be separated and treated as separate parameters.

[0119] (2) Majority voting is a method and condition for prioritizing multiple answers (answer groups) from multiple candidate LLM sites by evaluating and calculating the majority vote. This majority vote evaluation and calculation is performed by the priority processing AI 170 in Figure 3. The priority processing AI 170 compares multiple answers from multiple LLM sites and assigns a relatively high score to the majority answer and a relatively low score to the minority answer.

[0120] More specifically, for example, the process is as follows: The control unit 11 or the priority processing AI 170 first obtains a summary of each answer from multiple answers from multiple LLM sites. The number of answers from each LLM site is within the range of the "number of reference answers." The priority processing AI 170 compares the summaries of these multiple answers. In the comparison, answers whose summary content is in the majority and similar to others are given a relatively higher score, and answers whose summary content is in the minority are given a relatively lower score. Answers with similar content are given the same score.

[0121] (3) Reasoning Validity is a method and condition for prioritizing answers from LLMs by evaluating and calculating their validity. This validity evaluation and calculation is performed by the Priority Processing AI 170 shown in Figure 3. The Priority Processing AI 170 analyzes the basis of reasoning, evidence, citation sources, clarity of explanation, etc. for each answer from multiple LLM sites, and assigns higher scores to answers with higher validity.

[0122] [Example] Various settings are possible, not limited to the example settings of the setting items (rules, conditions, etc.) shown in Figure 5A. Figure 5B shows an overview of each setting example.

[0123] In setting example 1, the number of LLM sites used is 3 (e.g., LLM-A, B, C), the number of reference answers per site is 1, the number of displayed answers is 1, and the priority conditions are: credibility is ON (applied), majority voting is OFF (not applied), and validity is OFF (not applied).From the answers (e.g., A1, B1, C1) of the three LLMs (e.g., LLM-A, B, C), one answer, i.e., the answer with priority = 1 (e.g., A1), is selected and displayed based on the priority condition (credibility).

[0124] For the same target site as in Setting Example 1, it is also possible to set only majority voting to ON or only validity to ON as the priority condition.

[0125] In setting example 2, the number of LLM sites used is 3 (e.g., LLM-A, B, C), the number of reference answers per site is 1, the number of displayed answers is 1, and the priority conditions are: credibility ON, majority vote OFF, and validity ON. From the answers of the three LLMs, one answer, i.e., the answer with priority = 1 (e.g., A1), is selected and displayed based on the priority conditions (e.g., the overall evaluation of credibility and validity).

[0126] For the same target site as in Setting Example 1, it is also possible to set the trust rating and majority vote to ON, or the majority vote and validity to ON as priority conditions.

[0127] In setting example 3, the number of LLM sites used is 6 (e.g., LLM-A, B, C, D, E, F), the number of reference answers per site is 1, the number of displayed answers is 2, and the priority conditions are: credibility is OFF, majority voting is ON, and validity is OFF. Two answers (e.g., A1 and B1) are selected and displayed from the six answers from the six LLMs based on the priority condition (majority voting).

[0128] In setting example 4, the number of LLM sites used is 6 (e.g., LLM-A, B, C, D, E, F), the number of reference answers per site is 1, the number of displayed answers is 2, and the priority conditions are: credibility ON, majority vote ON, validity OFF. Two answers (e.g., A1 and B1) are selected and displayed from the six answers from the six LLMs based on the priority conditions (overall evaluation of credibility and majority vote).

[0129] In Figure 5C, setting example 5 has the following: number of LLM sites used = 3 (e.g., LLM-A, B, C), number of reference answers per site = 2, number of displayed answers = 1, and the priority conditions are trust level ON, majority voting ON, and validity ON. From up to two answers from each of the three LLMs (i.e., a maximum of six answers), one answer (e.g., A1) is selected and displayed based on the priority condition (an overall evaluation of trust level, majority voting, and validity).

[0130] In setting example 6, the number of LLM sites used is 3 (e.g., LLM-A, B, C), the number of reference answers per site is 3, the number of displayed answers is 3, and the priority conditions are trust level ON, majority vote ON, and validity ON. From up to three answers from each of the three LLMs (i.e., a maximum of nine answers), three answers (e.g., A1, B1, C1) are selected and displayed based on the priority conditions (an overall evaluation of trust level, majority vote, and validity).

[0131] FIG. 5D is an example of a screen display related to the user setting function, which shows a function that the user can select and apply from multiple settings. Item 5D1 allows the user to set a destination selection rule by selecting one from multiple options in a GUI such as a list box. A default destination selection rule (e.g., rule 1) is applied in advance, and the user can change it as needed. Item 5D2 allows the user to set an answer selection rule by selecting one from multiple options in a GUI such as a list box. A default answer selection rule (e.g., rule A) is applied in advance, and the user can change it as needed. Details of each rule can be confirmed in a separate GUI. Item 5D3 allows the user to set whether or not the response output device 1 of this system will automatically learn and update the user settings for the rules and conditions described above.

[0132] [Variations] As a modified example, a plurality of answers from a plurality of LLM sites may be selected and output for each priority condition. For example, the response output device 1 may select one answer from the plurality of answers based on the credibility condition, one answer based on the majority vote condition, and one answer based on the validity condition. The response output device 1 then outputs the three answers selected for each priority condition to the user along with information on the priority condition.

[0133] [If multiple answers are available for one LLM] In the basic configuration, the number of reference answers is set to 1, as in the example setting of Figure 5B. One answer is obtained for each question per LLM site (LLM server 2). However, this is not limited to this, and the number of reference answers may be set to multiple (2 or more), as in the example setting of Figure 5C, and multiple answers may be obtained per LLM.

[0134] The following is an example of how multiple answers can be obtained from one LLM (LLM Server 2) for one question prompt:

[0135] (1) There are cases where the LLM generates multiple answers (in other words, answer parts) for a question depending on conditional branching, different ways of thinking, etc. In general, one answer statement may contain multiple answer parts. For example, an answer statement may be something like, "If condition 1 (way of thinking 1) holds, then answer 1. If condition 2 (way of thinking 2) holds, then answer 2. If condition 3 (way of thinking 3) holds, then answer 3." In such cases, the response output device 1 may extract multiple answer parts from the answer statement, treat each answer part as an individual answer, and count them as the number of answers.

[0136] (2) Another example is when an LLM attempts to generate an answer to an input prompt multiple times, generating multiple answers as a result of each attempt.

[0137] [Condition setting process] FIG. 6 shows an example of the processing flow for setting and updating conditions in steps S1 and S2 of FIG. 4 by the response output device 1 (particularly the control unit 11). In step S601, the control unit 11 checks whether rules and conditions for user U1's use of the functions of this embodiment have already been set. Step S601 begins by checking the previous rule and condition settings based on the user U1's history and user settings. If this is the first time and conditions have not yet been set (NO), in step S602 the control unit 11 applies and sets the default conditions of the system to the user U1. If conditions have been set previously (YES), the process proceeds to step S603. In step S603, the control unit 11 displays the conditions already set for the user U1 (e.g., the settings shown in FIG. 5A) on the screen of the user interface 16. User U1 can check the conditions and settings on the screen.

[0138] In step S604, the control unit 11 confirms with the user U1 whether to change / update the condition / setting. Alternatively, the control unit 11 automatically determines whether to change / update the condition / setting without confirming with the user U1. For example, if the user U1 wants to change the rule / condition, he or she performs the necessary operation input. If the condition / setting is to be changed (YES), the process proceeds to step S605; if not (NO), the process ends. In step S605, the control unit 11 changes / updates the condition / setting. For example, the user U1 changes the setting value by operating the GUI on the screen. In step S606, the control unit 11 displays the changed / updated condition / setting on the screen. The user U1 can check the condition / setting on the screen.

[0139] [Display screen example] Figure 7A shows an example of a display screen on the display unit 13 of the response output device 1. This example screen corresponds to the setting shown in Figure 5A, particularly setting example 2 in Figure 5B. The target LLM sites are three LLM sites (A, B, C), and for each LLM site, the number of reference answers is 1, the number of displayed answers is 1, and one of the three answers is selected and displayed.

[0140] This screen has a user input field 701, a priority response field 702, a setting condition field 703, a priority condition field 704, an evaluation field 705, and the like.

[0141] A question prompt (text) based on user input information is displayed in the user input field 701. An example question is, "Please tell me the cheapest way to get from Yokohama Station to Ueno Station and how much it will cost." User U1 can input / update the question by pressing the "Input / Update" button.

[0142] The preferred answer field 702 displays the preferred answer (adopted answer) as the AI ​​(LLM)'s answer to the question prompt. The preferred answer is the answer selected as the answer to be output preferentially from among multiple answers (first group) from multiple candidate LLMs (LLM server 2) in accordance with the priority assigned based on the answer selection rules and conditions. The contents of the preferred answer field 702 are not displayed and show "Not displayed" until the text of the preferred answer is displayed. If the user wants to view the details of multiple answers (candidates), including the unadopted answer, they can do so by clicking the "View details" button. In this example, the number of displayed answers is 1, so only one answer field is provided in the preferred answer field 702.

[0143] The setting condition field 703 displays setting information related to the rules and conditions set in steps S1 and S2 described above. In this example, the setting condition field 703 displays specified information about the LLM sites to be used (target sites) that are candidates for the response source in the LLM site list (plurality of LLM servers 2 to which the response output device 1 can connect) as conditions and settings corresponding to the destination selection rules for bulk transmission. For each LLM site (LLM name), one with a check mark in the necessity item is designated as the destination. In addition, information such as the number of referenced responses and credibility (user evaluation) for each LLM site is also displayed. In this example, LLM-A, B, and C are set as destinations.

[0144] Detailed settings for the setting items shown in FIG. 5A can be made on a screen accessed by pressing the "Condition Settings" button.

[0145] User U1 can specify multiple LLMs (first group corresponding to the number of LLM sites used) to be the destination (candidates for the respondent), but this is not limited to this. If user U1 does not specify an LLM to be the destination, the response output device 1 will automatically select from the default setting LLM.

[0146] The priority condition column 704 displays the method and conditions for determining the priority of the priority answers, corresponding to the answer selection rules and conditions. For example, predefined methods and conditions such as (1) credibility, (2) majority vote, and (3) validity are provided, and user U1 can select and specify a combination of one or more of these as a priority condition. For example, user U1 can select the priority condition to apply from a list of priority condition options by operating a list box or the like. In this example, a priority condition (in other words, a mode) is specified that combines two conditions: credibility and validity.

[0147] The priority conditions (modes) for each combination are listed below. 1: Trustworthiness 2: Majority vote 3: Validity 4: Trustworthiness + majority vote 5: Credibility + Relevance 6: Majority vote + validity 7: Credibility + Majority + Validity

[0148] The question in this example is a question with a true answer based on quantitative information. The priority condition in this example is set to 5: credibility + validity. When there is a true answer, it is easy to evaluate credibility and validity.

[0149] The response output device 1 (particularly the control unit 11) uses the priority processing AI 170 to calculate a score, which is an evaluation value, for each of the multiple answers (first group) received and acquired from the LLM, based on the priority conditions (mode). In this example, since the evaluation is based on two perspectives (conditions), credibility and validity, the score is an overall score based on the overall evaluation of the two perspectives (conditions). One way to calculate this score (overall score) is to weight the evaluation value for each perspective (condition) and calculate the overall evaluation value. This score information is not generally output, but as will be described later, it is also possible to output this score information as detailed information.

[0150] The rating column 705 displays rating information about the LLM site that generated the priority answer (the answer selected based on the priority conditions) displayed in the priority answer column 702. In this example, this rating information is the rating points (or other numerical values ​​such as the credibility in FIG. 5A) as user rating information for the general user / user U1. This rating point is displayed together with the display of the priority answer, and is not displayed at the time of FIG. 4A.

[0151] Furthermore, in the evaluation field 705, after the prioritized answer is displayed in the prioritized answer field 702, user U1 can input a user evaluation and user comments (memo) for the prioritized answer (see FIG. 8A, which will be described later). The user evaluation may be selected from options such as "good" or "bad," or evaluation points may be entered. Before the prioritized answer is displayed in the prioritized answer field 702, the user evaluation and user comments cannot be entered.

[0152] Figure 7B is another example of a screen display. This example corresponds to setting example 6 in Figure 5C, where the target sites are three LLM sites (LLM-A, B, C), the number of reference answers per site is 3, and the number of displayed answers is 3. An example question in the user input field 701 is, "It is said that the universe is endless. What is the basis for this?"

[0153] In the priority condition column 704, credibility + majority vote + validity are set and displayed as priority conditions. The question in this example is a question for which the true answer is not determined using qualitative information. When the true answer is not determined, majority vote is useful.

[0154] In this example, there are a maximum of 3 x 3 = 9 possible answers. The preferred answer column 702 displays three preferred answers selected from these according to priority. The preferred answer column 702 has three answer columns corresponding to the three preferred answers. For example, the answer column marked [1] represents the column displaying the preferred answer with priority = 1. Initially, all answer columns are set to "not displayed." As answers are acquired, the answer with priority 1 is displayed in the answer column [1]. Similarly, answers with priority 2 and 3 are displayed in the answer columns [2] and [3].

[0155] [Answer display example] FIG. 8A shows an example of a display of preferred answers and the like when the question and conditions of FIG. 7A are set. When the number of displayed answers is 1, the answer with the highest score, priority level 1, selected from the three answers is displayed in the preferred answer column 702 as a single preferred answer [1]. This preferred answer [1] is, for example, the answer (B1) of LLM-B based on FIGS. 10A and 10C described below. In this example, this preferred answer [1] reads, "The cheapest way to get from Yokohama Station to Ueno Station is to use a transportation card to take the JR line. This is a bit of a trick, but by boarding the train at Yokohama Station, getting off at Shinagawa Station, passing through the ticket gate, and then boarding the train from Shinagawa Station to Ueno Station, you can get there for a total of ¥511 (¥303 + ¥208). It requires some effort to transfer, but it's ¥60 cheaper than the ¥571 direct fare."

[0156] The rating field 705 displays the rating points for the LLM site (LLM-B) that generated the preferred answer [1]. For example, the rating points are 4.25 points. These rating points are separate from the score of the answer. These rating points (user rating information) are reflected in the trust criteria. Furthermore, after viewing and confirming the preferred answer [1], user U1 can enter their own rating value in the user rating field of the rating field 705. This rating value is, for example, a rating point (e.g., a five-point scale from 1 point to 5 points), and can be selected from: very satisfied: 5 points, satisfied: 4 points, average: 3 points, dissatisfied: 2 points, and very dissatisfied: 1 point. The rating of this preferred answer is reflected as the rating (user rating information) for the LLM site that generated this preferred answer.

[0157] As a modified example, a Good button, a Bad button, or the like may be provided on the screen for each answer in the priority answer column 702. For example, when user U1 sees a certain priority answer and feels that it is a good answer for him / her, he / she presses the Good button.

[0158] As a modification, the evaluation column 705 may display the score of the preferred answer instead of the evaluation points of the LLM site.

[0159] Figure 8B shows an example of the display of preferred answers and other information for the question and condition settings in Figure 7B. In the preferred answer field 702, three preferred answers are displayed in order of priority from a maximum of nine possible answers. In this example, the preferred answer [1] with priority = 1 reads, "The edge of the universe is one of the mysteries that modern science has not yet fully explained. Here are some expert opinions: (1) Professor *** (*** University) *** (2) Mr. OOO (OOO University) *** (3) ******." The preferred answer [2] with priority = 2 reads, "Within the scope of our knowledge, there is no definitive answer regarding the edge of the universe. However, there are several hypotheses and theories. (1) Professor *** (*** University) *** (2) Professor XXX (XXX Observatory) *** (3) ******." The preferred answer [3] with priority = 3 reads, "Whether the universe has an edge is still not fully explained by modern science, but it is generally believed that the universe has no edge. ******." *** indicates an omission.

[0160] Based on Figures 10B and 10D described below, preferred answer [1] in Figure 8B is the first answer (A1) of LLM-A, preferred answer [2] is the second answer (A2) of LLM-A, and preferred answer [3] is the first answer (B1) of LLM-B.

[0161] In the example of Figure 8B, the priority answer column 702 does not display information about which LLM site each of the three priority answers belongs to. Separately, by operating the "display details" button, such information can be viewed as detailed information. As a variation, such information may be displayed in the priority answer column 702 from the beginning.

[0162] In the example of FIG. 8B, the evaluation field 705 shows a case where the user U1 can input an evaluation (evaluation points) for the answer selected by the user U1 from among the three priority answers.

[0163] [Answer selection process] 9 shows a detailed example of the answer selection process in step S9 of Fig. 4 in the response output device 1. The control unit 11 performs the answer selection process in step S9 shown in the figure for the answer group (first group) corresponding to the answer (response data) 202 received in the above-mentioned step S8.

[0164] In step S901, the control unit 11 selects and determines the priority condition (in other words, the mode, the priority determination mode) related to the answer selection rule based on the settings or the instruction of the user U1, etc. For example, in the example of Fig. 7A, the priority condition (mode) of credibility + validity is selected.

[0165] In step S902, the control unit 11 checks whether the processing (evaluation and calculation) of the selected priority condition (mode) requires processing by an AI. In this embodiment, the priority processing AI 170 (FIG. 3) is used as this AI. In this embodiment, AI processing (priority processing AI 170) is required for the majority vote and the evaluation and calculation of validity, but AI processing (priority processing AI 170) is not required for the evaluation and calculation of creditworthiness. In this embodiment, the priority processing AI 170 is not required for the evaluation and calculation of creditworthiness because the control unit 11 can calculate the creditworthiness using relatively simple processing from the user evaluation information of general users or user U1. However, whether the priority processing AI 170 (in other words, an AI inside or outside the response output device 1) is required is determined depending on the type and content of each priority condition.

[0166] If AI processing is required (YES), proceed to step S904 or S905, which is the step corresponding to the required AI processing. Step S904 is the evaluation and calculation of the majority vote (AI processing), and step S905 is the evaluation and calculation of the validity of the inference (AI processing). If AI processing is not required (NO), that is, if the evaluation and calculation processing of the credibility is to be performed, proceed to step S903. Note that in the case of a priority condition that is a combination of two or more conditions, the processing of steps S903, S904, and S905 corresponding to the priority conditions can be executed sequentially or in parallel.

[0167] In step S903, the control unit 11 calculates the credibility of the LLM site that generated the answer by referring to the evaluation information of general users and / or user U1. If the credibility has already been calculated, it is sufficient to simply refer to that credibility. As a result of the processing in step S903, the control unit 11 obtains a credibility score (evaluation value) for each answer. In step S904, the control unit 11 uses a majority voting module in the priority processing AI 170 to compare multiple answers (first group) and determine a majority vote, and obtains a majority vote score (evaluation value) for each answer. In step S905, the control unit 11 uses a validity module in the priority processing AI 170 to determine the validity of each answer in the multiple answers (first group), and obtains a validity score (evaluation value) for each answer.

[0168] In step S906, the control unit 11 calculates a total score for each answer using the scores obtained as a result of the evaluations in steps S903 to S905 according to the priority condition. If there is one priority condition, the score for that priority condition may be used as the total score. If there is a combination of two or more priority conditions, the total score may be calculated by weighting the scores for each priority condition. For example, in the case of a combination of credibility, majority vote, and validity, the scores for credibility, majority vote, and validity may be X, Y, and Z, respectively, and the weighting coefficients for each may be A, B, and C. The control unit 11 may calculate the total score S, which is the overall evaluation value, using a formula such as S=A×X+B×Y+C×Z. For example, in the case of a priority condition of credibility+validity, A=0.5, B=0, C=0.5, and S=(X+Z) / 2 may be used.

[0169] In step S907, the control unit 11 assigns and determines priorities (or priorities) to the multiple answers based on the total score of each answer. For example, the control unit 11 assigns priorities = 1, 2, 3, ... in descending order of total score.

[0170] In step S908, the control unit 11 selects and determines a priority answer to be output by selecting an answer with a high priority within the range of the number of answers to be displayed based on the priority of the answers.

[0171] [Example of evaluation results] Figure 10A shows an example of the scores for the evaluation results when two items are evaluated based on the priority criteria (trustworthiness + validity) in Figure 7A. Each LLM (LLM-A, B, C) has a trustworthiness score, a majority vote score (left blank in Figure 10A because majority votes are not used), a reasoning validity score, and an overall score. The trustworthiness score is based on the evaluation information of the general user / user U1 regarding the LLM site. Trustworthiness is an evaluation value for each LLM site. For example, out of a maximum score of 100, LLM-A is 90 points, LLM-B is 50 points, and LLM-C is 30 points. The reasoning validity score is 0 points for LLM-A, 100 points for LLM-B, and 30 points for LLM-C for each answer (assuming answer 1) for each LLM site. The overall score is, for example, the sum of the scores for the three items (trustworthiness, majority vote, and validity). In this example, LLM-A is 90 points + 0 points = 90 points, LLM-B is 50 points + 100 points = 150 points, and LLM-C is 30 points + 30 points = 60 points.

[0172] Priorities can be assigned based on the overall score. In this example, in descending order of overall score, priority = 1 is assigned to the LLM-B answer, priority = 2 is assigned to the LLM-A answer, and priority = 3 is assigned to the LLM-C answer. If the number of displayed answers is 1, the LLM-B answer with priority = 1 is selected as the first prioritized answer. In another example, if the number of displayed answers is 2, the LLM-A answer with priority = 2 is selected as the second prioritized answer.

[0173] Figure 10B shows an example of the scoring results for an evaluation based on three items corresponding to the priority criteria (trustworthiness + majority vote + validity) in Figure 7B. Each LLM site has a score for up to three answers (Answer 1, Answer 2, and Answer 3). Trustworthiness is an evaluation value for the LLM site, and up to three answers have the same score. Based on these scores, each answer has an overall score.

[0174] In this example, the credibility scores are the same as in Figure 10A. Regarding the majority vote scores, for LLM-A, Answer 1 is 80 points, Answer 2 is 50 points, and Answer 3 is 0 points. For LLM-B, Answer 1 is 80 points, Answer 2 is 50 points, and Answer 3 is 50 points. For LLM-C, when two answers are obtained, Answer 1 is 80 points and Answer 2 is 50 points. Answer 1 for LLM-A, Answer 1 for LLM-B, and Answer 1 for LLM-C are similar in gist (indicated by parentheses (a)) and were the majority (three answers), so they received a relatively high score of 80 points. LLM-B Answer 2 and LLM-C Answer 2 have similar main points (indicated by bracketed (b)) and are split into two parts. LLM-A Answer 1 and LLM-B Answer 2 have similar main points (indicated by bracketed (c)) and are split into two parts. These splits are smaller than the majority (three parts) and are each worth 50 points. LLM-A Answer 3 has only one main point (indicated by bracketed (d)) and is worth 0 points.

[0175] For relevance scoring, LLM-A Answer 1 is worth 80 points, Answer 2 is worth 70 points, and Answer 3 is worth 50 points. LLM-B Answer 1 is worth 80 points, Answer 2 is worth 60 points, and Answer 3 is worth 70 points. LLM-C Answer 1 is worth 70 points, and Answer 2 is worth 60 points.

[0176] For example, answer 1 for LLM-A is worth 90 points + 80 points + 80 points = 250 points. Answer 2 is worth 90 points + 50 points + 70 points = 210 points. Answer 3 is worth 90 points + 0 points + 50 points = 140 points. Similarly, the three answers for LLM-B are worth 210 points, 160 points, and 170 points, respectively. The two answers for LLM-C are worth 180 points and 140 points, respectively.

[0177] In this example, the total score is calculated by adding up the scores of the three items, but this is not limiting and the total score may be normalized to a maximum of 100 points. When normalized, for example, 250 points becomes 250 ÷ 3 ≒ 83 points.

[0178] When selecting three priority answers from these eight answers corresponding to the number of displayed answers = 3, the priorities are 1, 2, 3 in descending order of overall score, and you can select LLM-A Answer 1, LLM-A Answer 2, and LLM-B Answer 1.

[0179] In this example (Figures 10B and 10D), Answer 2 in LLM-A and Answer 1 in LLM-B have the same total score (210 points). Therefore, these two answers may conceptually have the same priority. However, in this example, in order to distinguish them as different priority answers by setting a hierarchy on the screen display (e.g., Figure 8B), these two answers are assigned different priorities (2, 3). In this case, any method can be used to assign priorities between answers with the same score.

[0180] Figure 10C is a table of priorities determined in response to the evaluation results of Figure 10A. Answer B1 (LLM-B's Answer 1), which has a priority of 1, is the preferred answer [1].

[0181] Figure 10D is a table of priorities determined in response to the evaluation results of Figure 10B. Answer A1 (LLM-A answer 1) with priority = 1 becomes preferred answer [1], answer A2 (LLM-A answer 2) with priority = 2 becomes preferred answer [2], and answer B1 (LLM-B answer 1) with priority = 3 becomes preferred answer [3].

[0182] [Answer display processing] 11 shows a detailed example of the answer display process in step S10 of FIG. 4 in the response output device 1. In step S1101, the response output device 1 displays the priority answer selected in step S9 on the screen of the display unit 13 (FIG. 1) of the user interface 16. A specific example is shown in FIG. 8A, etc. When audio output is used, the response output device 1 outputs the audio of the priority answer by audio from the audio unit 14.

[0183] In step S1102, the response output device 1 makes a selection and decision from the following options in response to a user input operation or the like. (1) Detailed display information (2) Continue to enter the next prompt (3) Changes to Terms (4) Evaluation of responses (5) History storage and learning (6) End

[0184] In this embodiment, in order to facilitate a smooth conversational interaction between user U1 and the AI, in step S1102, GUIs that enable input operations corresponding to the above options (1) to (6) are displayed together on the same screen. This allows an option to be selected easily with a single operation, and does not require user U1 to perform input operations for each option. In the example screen of FIG. 8A, option (1) can be selected using the detail display button in the preferred answer field 702. option (2) can be selected using the input / update button in the user input field 701. option (3) can be selected using the condition setting button, the set condition field 703, or the preferred condition field 704. option (4) can be selected in the evaluation field 705. option (5) can be selected using the history / learning button. option (6) can be selected using the exit button.

[0185] The present invention is not limited to this, and various other GUIs are applicable. In a modified example, a GUI including multiple options such as (1) to (6) (for example, a pop-up 1510 in FIG. 15) may be displayed.

[0186] If (1) is selected, proceed to step S1103. The response output device 1 displays primary detailed information about the prioritized answer on the screen. The primary detailed information display displays a list of the scores of each answer in each LLM in the prioritized answer of the number of displayed answers (see FIGS. 12A and 12B, which will be described later). As the primary detailed information, for example, information on the prioritized answer in FIG. 10C or 10D may be displayed. In step S1104, whether to further detail the display is selected according to a user input operation, etc. If further detailing is desired (YES), proceed to step S1105. In step S1105, the response output device 1 displays secondary detailed information about the answer on the screen. The secondary detailed information display displays information on each answer in each LLM other than the prioritized answer of the number of displayed answers, i.e., information on answers with low priority. From step S1104 or S1105, return to step S1102.

[0187] Alternatively, the primary detailed information and the secondary detailed information may be displayed together by pressing the details button once.

[0188] If (2) is selected, the process returns to step S3, where a prompt for the next question is created based on the user's input operation. If (3) is selected, the process returns to step S2, where the conditions are changed / updated. The conditions include the aforementioned destination selection rules and conditions and answer selection rules and conditions, and either of them may be changed / updated.

[0189] If (4) is selected, the process proceeds to step S1106, where user U1 inputs an evaluation in the evaluation field 705 (e.g., FIG. 7A) on the screen. Based on the input evaluation, control unit 11 updates the evaluation value (the aforementioned credibility) of the LLM site. For example, control unit 11 updates the credibility of the LLM site so that the evaluation points input by user U1 are reflected in the credibility. If user U1 does not input an evaluation, the evaluation value is not updated.

[0190] If (5) is selected, the process proceeds to step S1107, where the control unit 11 stores history information related to the function of this embodiment in memory. The history information is history information related to a series of processes from setting conditions, creating prompts, and sending them, to receiving, selecting, and outputting answers. The history information includes, for example, prompts, answers, destination selection rules, answer selection rules, and answer evaluation results (scores and user evaluation information). The control unit 11 learns about the rules and conditions based on the history information and updates the rules and conditions appropriately according to the learning results. If (6) is selected, use of the function of this embodiment ends.

[0191] The system may be configured to automatically and always save and learn the history information (5) as a default setting. Alternatively, the system may have fixed settings for rules and conditions. Regarding the saving of history information, all related information may be saved, but this is not limited to this, and only selected information may be saved. For example, it is possible to save only questions and preferred answers.

[0192] [Details] FIG. 12A shows an example of the primary detailed information display screen in step S1103. This example corresponds to the examples in FIGS. 7A and 8A. User input fields and the like are not shown. When the detailed information display button in FIG. 8A is pressed, a primary detailed information field 1201 like that in FIG. 12A is displayed. In the primary detailed information field 1201, information like the evaluation result table in FIG. 10A is displayed as primary detailed information. User U1 can check the scores and other information for answers at each LLM site, including answers other than the preferred answer. The screen in FIG. 12A has a secondary detailed information display button and a return to preferred display button. When the return to preferred display button is pressed, the screen returns to the screen display shown in FIG. 8A (in other words, the normal display). Furthermore, when the secondary detailed information display button is pressed, secondary detailed information (secondary detailed answers) like that shown in FIG. 13A is displayed.

[0193] Similarly, Figure 12B shows an example of a screen displaying primary detailed information corresponding to the examples of Figures 7B and 8B. When the detail display button in Figure 8B is pressed, a primary detailed information field 1201 like that shown in Figure 12B is displayed. In the primary detailed information field 1201, information like the evaluation result table in Figure 10B is displayed as primary detailed information. User U1 can check the scores and other information for answers on each LLM site, including answers other than the preferred answer. The screen in Figure 12B has a secondary detailed information display button and a return to preferred display button. When the return to preferred display button is pressed, the screen returns to the screen display shown in Figure 8B (in other words, the normal display). Furthermore, when the secondary detailed information display button is pressed, secondary detailed information (secondary detailed answers) like that shown in Figure 13B is displayed.

[0194] FIG. 12C is a modification of FIG. 12A, and in the primary detailed information field 1201, information including the priority order as in FIG. 10C is displayed.

[0195] FIG. 14B is a modification of FIG. 12B, and in the primary detailed information field 1201, information including the priority order as shown in FIG. 10D is displayed.

[0196] 13A and 13B show examples of secondary detailed display screens. This example corresponds to the examples of FIGS. 7A and 8A. FIG. 13A shows an initial display of, for example, answer B1 with priority level 1 in the foreground, and FIG. 13B shows an initial display of, for example, answer A1 with priority level 2 in the foreground after transition from FIG. 13A. When user U1 requests secondary detailed display by operating the aforementioned secondary detailed display button, the response output device 1 displays, for example, a secondary detailed information answer field 1301 as shown in FIG. 13A on the screen. This field 1301 displays information on multiple answers, including answers other than the priority answer and not selected, by overlapping them, for example, in the front-to-back direction on the screen. In this example, a separate answer field 1303 is provided for each of the multiple answers.

[0197] In this example, a portion of the screen first has a table 1302 of detailed information similar to the primary detailed information shown in FIG. 12A or 12C. In this example, this table 1302 is a table including priorities, similar to FIG. 12C. In the initial state of FIG. 13A, multiple answers (corresponding multiple answer columns 1303) are arranged in order of priority, overlapping one another. Answer B1, which is the preferred answer [1] with priority = 1, is displayed in the foreground, answer A1 in the second place, and answer C1 in the third place at the back. All of the answer B1 in the foreground is visible. In this example, the three answer columns 1303 are arranged diagonally from the top left to the bottom right, but this is not limited to this. In this initial state, user U1 can confirm the contents of answer B1, which is the preferred answer [1].

[0198] User U1 can select a desired answer by looking at the information in table 1302 or multiple answer columns 1303. For example, user U1 can select a row of a desired LLM for which the answer is to be viewed from table 1302. For example, if user U1 wants to view answer A1 of LLM-A with priority level = 2, user U1 selects that row. The response output device 1 changes the screen display state as shown in FIG. 13B so that the answer column 1303 of the selected answer can be easily viewed.

[0199] 13B, the selected row in the table 1302 is switched to a prominent display, for example, a display in a predetermined color, to indicate the selected state. The response output device 1 switches the display state in the secondary detailed answer column 1301 so that the answer of the selected row (for example, answer A1) is displayed in the foreground answer column 1303. In this example, the positions of the three answer columns 1303 remain the same, but the answer column 1303 of the selected answer A1 is moved to the foreground, and the answer columns 1303 of the other answers B1 and C1 are moved to be behind the answer column 1303 of the selected answer A1.

[0200] Another method of operation is for user U1 to click or otherwise select the visible part of answer field 1303 for answer A1 that is behind in Figure 13A, so that answer field 1303 for answer A1 is displayed in the foreground.

[0201] As another example of display control, when user U1 selects answer A1, the answer column 1303 for answer A1 may be moved to the position (top left) of the answer column 1303 for answer B1 and displayed in the foreground, and the answer column 1303 for answer B1 may be moved to the position (center) of the answer column 1303 for answer A1 and displayed behind it. In other words, the display order and positions of the answer columns 1303 for multiple answers may be rearranged.

[0202] Figures 13C and 13D are similar screen examples of secondary detailed display corresponding to the examples of Figures 7B and 8B, and are outlined similarly to Figures 13A and 13B. Figure 13C shows the initial state, with answer A1 from LLM-A, which has priority level 1, displayed in the foreground answer column 1303. In this example, answer columns are provided for each answer group (number of reference answers = 3) on the LLM site. For example, answers A1, A2, and A3 from LLM-A are displayed as a single group in order of priority in a single answer column 1303. The circled numbers 1, 2, and 3 correspond to answers A1, A2, and A3. Answers A1, A2, and B1, which are priority answers [1], [2], and [3] with a display answer count of 3, are displayed in a specific color to make them stand out. A detailed information table 1302 similar to that shown in Figure 12D is also provided. 13C, for convenience of explanation, the answer column 1303 is shown overlapping the detailed information table 1302, but this is not limiting. In the initial state of FIG. 13C, the user U1 can easily check the answer A1, which has priority level 1.

[0203] User U1 selects a desired answer, for example, the row of answer B1 of LLM-B, in the detailed information table 1302. In this case, the response output device 1 switches the state so that the answer column 1303 of the LLM-B group including answer B1 of LLM-B is displayed in the foreground, as shown in FIG. 13D. In the state of FIG. 13D, user U1 can easily check answer B1, which has priority level 2. In another example, when user U1 selects answer C1 of LLM-C, the answer column 1303 of the group including answer C1 is switched to be in the foreground.

[0204] In this example, one or more answers for each LLM site are grouped together to form an answer field 1303, and the answer fields 1303 for each group are displayed overlapping one another, but the display method is not limited to this and can be used.The answer field 1303 for each LLM site group may also be omitted (see, for example, Figure 16 described below).

[0205] [Examples of answer evaluation] A specific example of answer evaluation will be explained below. Using the examples of setting example 2 in Fig. 5B, Fig. 7A, Fig. 8A, Fig. 10A, Fig. 10C, Fig. 13A, and Fig. 13B, we will explain the case where answers A1, B1, and C1 exist and answers B1, A1, and C1 have priority levels 1, 2, and 3.

[0206] LLM-A's answer A1 was, "The cheapest way to get from Yokohama Station to Ueno Station is to take the train using a transportation card. If fare is a priority, take the Tokyu Toyoko Line from Yokohama Station to Nakameguro Station, then transfer to the Tokyo Metro Hibiya Line to Shinagawa Station for 518 yen. This route requires one transfer, but it is the cheapest option."

[0207] LLM-A's answer B1 was, "The cheapest way to get from Yokohama Station to Ueno Station is to use a transportation card to take the JR line. It's a bit of a secret trick, but by boarding the train at Yokohama Station, getting off at Shinagawa Station, going through the ticket gate, and then boarding the train from Shinagawa Station to Ueno Station again, you can get there for a total of ¥511 (¥303 + ¥208). It takes some effort to transfer, but it's ¥60 cheaper than the direct fare of ¥571."

[0208] LLM-A's answer C1 was, "The cheapest way to get from Yokohama Station to Ueno Station is to use the JR line with a transportation card. From Yokohama Station, it costs 580 yen to take the Tokaido Line, Yokosuka Line, or Keihin-Tohoku Line to Shinagawa Station. This is a direct and quick route, so I recommend it."

[0209] In this case, the evaluation score for priority conditions is calculated as follows, for example. Regarding "appropriateness," LLM-A's answer A1 is more expensive than LLM-B's answer B1, even though it involves a transfer. Therefore, the appropriateness of answer A1 cannot be evaluated and it is given a score of 0. LLM-B's answer B1 is the cheapest route desired by the user, even though it involves a transfer, so the appropriateness of answer B1 is given a score of 100. LLM-C's answer C1 is more expensive than the other answers, but it focuses on the convenience of being able to travel directly and the time savings (a new perspective), so it is given an additional score of 30.

[0210] In the case of setting example 6 in FIG. 5C, and the examples in FIGS. 7B, 8B, 10B, 10D, 13C, and 13D, the following applies. We will explain the case where there are eight answers and answers A1, A2, and B1 have priorities of 1, 2, and 3.

[0211] In Figures 10B and 10D, the evaluation scores for the priority conditions are calculated as follows: In the "majority vote" evaluation calculation, all eight answers are compared and a majority vote is made. For example, answers A1, B1, and C1 share the same gist and are the majority (three answers), so they are assigned 80 points. Answers A1, B1, and C1 introduce several hypotheses without a conclusion. The next two majority answers are answers A2 and B3, and answers B2 and C2, each of which are assigned 50 points. Answers A2 and B3 present the idea that there is no edge to the universe, while answers B2 and C2 present the idea that there is an edge to the universe. Answer A3 is the only one of the answers, so it is assigned 0 points. Answer A3 does not directly answer the question and is deemed to have low validity, so it is assigned 50 points.

[0212] [Modification: Screen display] The present invention is not limited to the display control example shown in Fig. 13A etc. As a modified example, as shown in Fig. 14, in the priority answer column 702 of the screen, for example, in the detailed answer column 1401, multiple pieces of answer information may be displayed in parallel in order of priority. Alternatively, as a modified example, as shown in Fig. 15, some of the multiple pieces of answer information may be displayed on the screen, and other pieces of answer information may be additionally displayed in response to a user operation. Alternatively, as a modified example, as shown in Fig. 16, multiple answers may be displayed in a matrix.

[0213] FIG. 14 shows an example of a detailed answer display screen corresponding to the examples of FIGS. 7A and 8A. For example, when the detailed display button described above is pressed, detailed answer information is displayed in a detailed answer field 1401. In this detailed answer field 1401, all of the multiple answers in the answer group (first group) are displayed in parallel, for example, from top to bottom, in order of priority / score. The detailed answer field 1401 includes an answer field 1402 for each answer. The answer field 1402 may also display information such as the LLM name, answer ID, priority, or score. Operations such as saving and deleting may also be possible for each answer. For example, when the save button is pressed, the answer information is saved in a file or the like with a name, and when the delete button is pressed, the answer field 1402 for that answer may be hidden. On this screen, user U1 can check the answers in order of priority, from top to bottom.

[0214] FIG. 15 shows another example of a display screen. This example corresponds to the examples of FIGS. 7B and 8B. In the prioritized answer column 702, initially, the number of displayed answers is 3, and answer columns 1501 for three prioritized answers [1], [2], and [3] are displayed in parallel, for example, from top to bottom, according to priority. When user U1 presses the detailed display button once, for example, table 1502 is displayed as primary detailed information. This table 1502 contains information similar to that shown in, for example, FIGS. 10D and 12D. Table 1502 also has check boxes in the "Display" column. Answers in checked rows are displayed in the answer column 1501, and answers in unchecked rows are not displayed. Initially, answer columns 1501 for prioritized answers [1], [2], and [3] with priority levels 1, 2, and 3 are automatically displayed.

[0215] User U1 checks the "display" checkbox of the desired row. For example, assume that the row of answer C1 with priority = 4 is checked as shown in the figure. In this case, the control unit 11 displays answer C1 as an answer column 1501 for answer [4] below the three answer columns 1501 that are already displayed. This allows user U1 to check information on answers that were not initially selected.

[0216] As another operation, suppose that user U1 unchecks the row of answer A2 with priority level 2 and for which priority answer [2] has already been displayed. In this case, the control unit 11 turns off the answer column 1501 for priority answer [2] of answer A2. Alternatively, a display on / off button may be provided for each individual answer column 1501.

[0217] As other examples of the GUI, a "Show next answer" button 1503, a "Show fewer answers" button 1504, etc. may be provided. When the user U1 presses the "Show next answer" button 1503, the control unit 11 additionally displays an answer (e.g., answer B3) that corresponds to the next priority among the undisplayed answers in relation to the already displayed answers (e.g., priority answers [1], [2], [3] and answer [4]) below the already displayed answer column 1501. Also, when the user U1 presses the "Show fewer answers" button 1504, the control unit 11 switches off (hides) the answer column 1501 of the answer with the lowest priority (i.e., the answer column 1501 at the bottom) among the already displayed answer columns 1501.

[0218] As another example of a GUI, as described above (FIG. 11), a GUI that displays multiple options together, such as a pop-up 1510, may be provided. User U1 can select an option from the pop-up 1510.

[0219] 16 shows another example of a display screen. In this example, in the priority answer column 702, multiple candidate LLM sites (e.g., LLM-A, B, C) are arranged in parallel horizontally, and multiple answers (the number of reference answers) from each LLM site are arranged in parallel vertically, providing, for example, a 3x3 individual answer column 1601 (in other words, areas) in a matrix. In this matrix, the response output device 1 automatically displays answers with high priority (e.g., A1, A2, B1) in parallel up to the number of answers to be displayed as priority answers, and leaves other answers with lower priority that exceed the number of answers to be displayed blank (e.g., grayed out).

[0220] If user U1 wants to view other answers, user U1 operates, for example, by clicking on the desired blank answer column 1601. In response to this operation, the response output device 1 additionally displays the corresponding answer in that answer column 1601. For example, when user U1 operates the answer column 1601 at the position [C1] in the figure, answer C1 is displayed in that answer column 1601.

[0221] [Variation: Example of control when there is a time lag in receiving multiple responses] When the response output device 1 receives and acquires multiple responses from multiple candidate LLM servers 2 in response to a batch transmission (step S8 in FIG. 4), there may be a significant time difference between the reception of each response. This time difference may occur depending on the type, performance, load, communication status, etc. of each LLM server 2. The response output device 1 may display selected responses within the number of display responses at each point on the time axis depending on the response reception status.

[0222] As a modified example, an example of display control on the time axis in this case will be illustrated using Figure 17. The example of the question is the same as that in Figure 7A described above. The candidate LLM sites are LLM-A, B, and C, and the set number of displayed answers is 1. After sending the questions in bulk in step S4 of Figure 4, the response output device 1 waits for a group of answers in step S8. For example, at time 1, it is assumed that answer A1 is first received from LLM site A. At time 2, it is assumed that answer B1 is received from LLM site B. At time 3, it is assumed that answer C1 is received from LLM site C.

[0223] At time point 1, the response output device 1 only has answer A1, so it displays answer A1 in the answer column 1701 on the screen regardless of the score. Next, at time point 2, the response output device 1 compares answer A1 with the received answer B1, and displays one answer selected in accordance with the priority in the answer column. For example, suppose answer A1 has a score of 50 points and answer B1 has a score of 40 points. The response output device 1 selects answer A1, which has the higher score, and because answer A1 has already been displayed, it maintains the display state of answer A1.

[0224] Next, at time point 3, the response output device 1 compares answer A, answer B, and the received answer C, and displays one answer selected in accordance with the priority judgment in the answer column. For example, assume that answer C1 has a score of 60 points. Therefore, the response output device 1 selects answer C1 with the highest score, and since the number of output answers is 1, displays answer C1 in the answer column 1701 instead of answer A1 that has already been displayed. In this way, answers are selected in accordance with the priority at each time point and displayed within the range of the number of output answers.

[0225] When applying such an example of display control, even if not all of the candidate answers have been displayed, by displaying one of the answers, the time during which the answer is not displayed and the waiting time of the user U1 can be reduced.

[0226] [Other display examples] FIG. 18 shows another example of a screen display, in which the priority answer column 702 displays a situation in which the acquisition of an answer from a certain LLM is delayed. In this example, answers A1, B1, and C1 are acquired from LLM-A, B, and C, respectively, and the number of displayed answers is 1. For example, at a certain point in time, answer A1 from LLM-A and answer C1 from LLM-C have been acquired, while answer B1 from LLM-B has not yet been acquired. In this example, there is an answer column 1801 for each LLM, and the three answer columns are simply arranged in the order of LLM-A, B, and C. Answer A1 from LLM-A and answer C1 from LLM-C have been acquired, and their scores are 50 and 60 points, respectively. Based on the priority, answer C1 is provisionally assigned priority 1 and answer A1 is provisionally assigned priority 2 at that time.

[0227] The response output device 1 displays the answer C1 in the answer column 1801 of LLM-C, and indicates that the answer C1 is a preferred answer with a priority of 1. In this example, the answer column 1801 of LLM-C is displayed with a predetermined color or an image such as a ribbon to indicate that the answer C1 is a preferred answer with a priority of 1. The answer column 1801 of LLM-A is displayed to indicate that the answer A1 has been acquired but is not a preferred answer.

[0228] The response output device 1 displays in the answer column 1801 of LLM-B a message indicating that answer B1 has not been obtained, and also displays the estimated time until answer B1 can be obtained based on past history information. If the history information stores actual results such as the time required to obtain an answer in a question-and-answer exchange with LLM-B, it is possible to predict the average time, etc. from the actual results.

[0229] Thereafter, when the response output device 1 receives and acquires answer B1, the priority may change depending on the score of answer B1. The response output device 1 updates the display in the answer column 1801 depending on the changed priority. For example, if answer B1 has the highest score, such as 70 points, answer B1 will have priority = 1 and answer C1 will have priority = 2. The response output device 1 displays answer B1 in the LLM-B answer column 1801 as a prioritized answer with priority = 1, and turns off the answers in the other answer columns 1801. Furthermore, if the response output device 1 is unable to receive and acquire answer B1 even after waiting for a certain amount of time, the response output device 1 may stop accepting answer B1 at that point and display in the LLM-B answer column 1801 that answer B1 could not be acquired.

[0230] Furthermore, the response output device 1 may control the standby time after the simultaneous transmission process based on the estimated time required for each LLM from sending a question to receiving a response. The standby time from each LLM may be the same or different.

[0231] [Other variants (1)] Other possible variations include the following: The response output device 1 sets one LLM server 2 as the destination and source of questions and answers, and receives and acquires multiple answers from this LLM server 2 within the set number of reference answers. Alternatively, if multiple answer parts can be obtained from one answer from this LLM server 2 by separating the subject matter, the response output device 1 acquires the multiple answer parts. The response output device 1 then prioritizes the multiple answers / answer parts according to the answer selection rules / conditions, and outputs them to user U1 within the number of output answers.

[0232] [Other variations (2)] In this embodiment, a case has been described in which an answer selected within the range of the number of displayed answers is displayed as a preferred answer even if the total score of the candidate answer is low. As a variation, depending on the total score of the answer, if the total score of the answer is lower than, for example, a threshold, the answer may not be selected or displayed even if it is within the range of the number of displayed answers. In this case, the response output device 1 may output to the user U1 that the preferred answer will not be displayed because the evaluation score of the answer was low. When the user U1 sets the score threshold, the number of answers to be displayed can be adjusted depending on the credibility desired by the user U1, etc.

[0233] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. Except for essential components, components can be added, deleted, or replaced in each embodiment. Unless otherwise specified, each component can be singular or plural. A combination of each embodiment and its variations is also possible. [Explanation of symbols]

[0234] 1...Response output device (AI response output device), 2...LLM server, 201...Prompt (request data), 202...Answer (response data), U1...User.

Claims

1. A response output device that outputs a response related to artificial intelligence (AI), A prompt, which is the same question sentence created based on the user's input information, is sent in bulk to a first group of multiple large-scale language models (LLMs) that are candidate answer sources; receiving and acquiring a plurality of responses from the plurality of LLMs in the first group, the responses including a response from each LLM; Response output device.

2. 2. The response output device according to claim 1, automatically selecting the first group of the plurality of LLMs based on a setting; Response output device.

3. 2. The response output device according to claim 1, outputting information on a plurality of LLMs available as the candidates to the user, and selecting the plurality of LLMs in the first group based on at least one designation from the information on the plurality of LLMs by the user; Response output device.

4. 2. The response output device according to claim 1, For a first answer group that is the plurality of answers received from the plurality of LLMs of the first group, prioritizing the answers based on a set rule, selecting answers to be output from the first answer group in accordance with the priorities so that the number of output answers is less than the number of answers in the first answer group, and outputting the selected answers to the user. Response output device.

5. 5. The response output device according to claim 4, The credibility of the LLM of the respondent is used as one of the conditions constituting the rule. Response output device.

6. 6. The response output device according to claim 5, The credit rating is Paid or free sites? General LLM or Specialized LLM? Evaluation information of the LLM by a general user or the user; is calculated using at least one of Response output device.

7. 7. The response output device according to claim 6, a user interface that allows the user to evaluate the answer and input evaluation information based on the result of the user checking the output answer; updating the creditworthiness of the LLM based on the evaluation information by the user; Response output device.

8. 5. The response output device according to claim 4, a majority vote in the first group of answers is used as one condition for configuring the rule; Response output device.

9. 9. The response output device according to claim 8, The majority vote calculation is performed using external or internal AI. Response output device.

10. 5. The response output device according to claim 4, The validity of the inference about the answer is used as one of the conditions constituting the rule. Response output device.

11. 11. The response output device according to claim 10, The calculation of the validity of the inference is performed using external or internal AI. Response output device.

12. 5. The response output device according to claim 4, The conditions constituting the rule are as follows: The credibility of the respondent's LLM; and a majority vote on the first set of responses; the validity of the inference about the answer; determining the priority based on a combination of two or more of the trustworthiness, the majority vote, and the validity; Response output device.

13. 13. The response output device according to claim 12, calculating an evaluation value for each of the conditions of the combination, calculating a comprehensive evaluation value from the calculated evaluation values, and determining the priority order based on the comprehensive evaluation value; Response output device.

14. 5. The response output device according to claim 4, outputting detailed information about the priority order when the output answer was selected to the user in response to an operation by the user; Response output device.

15. 5. The response output device according to claim 4, Regarding the output answers, in response to an operation by the user, an answer that was not selected from the first answer group is output to the user. Response output device.

16. 5. The response output device according to claim 4, maintaining historical information regarding a series of processes from issuing the prompt to receiving, selecting, and outputting the answer; learning the rules for assigning the priorities based on the history information and updating the rules; Response output device.

17. 5. The response output device according to claim 4, The plurality of LLMs in the first group that are candidates for the response source include a local LLM provided inside the response output device. Response output device.

Images