Answer program, answer method and answer system

A hybrid local-server model for generative AI services optimizes computing resource use by employing local language models with quality evaluation and server-side processing, addressing the resource and quality challenges in local environments.

JP7588752B1Active Publication Date: 2024-11-22FIXER

Patent Information

Application Number
JP2024124787
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

Generative AI services using large language models (LLMs) require significant computing resources, which is not feasible in local environments like edge devices and local PCs, leading to inferior generation quality compared to server-based LLMs.

Method used

A hybrid approach using a first local language model and a second local language model, with the latter evaluating the quality of the first model's responses. If the quality is insufficient, the request is sent to a server-side language model for processing, ensuring high-quality responses while optimizing resource usage.

Benefits of technology

This method ensures high-quality responses while efficiently utilizing computing resources, balancing local and server-side processing to maintain generation quality without excessive resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007588752000001_ABST
    Figure 0007588752000001_ABST
Patent Text Reader

Abstract

A reply program etc. is provided that makes optimal use of computing resources and provides an appropriate reply to a user request. [Solution] The answer program causes the answer device to execute the following answer method. The request receiving step receives a request from a user. The first answer generating step causes a first local language model to generate a first answer based on the request. The determining step determines whether or not to present the first answer to the user by having a second local language model evaluate the first answer. The request sending step sends a request to a server when it is determined that the first answer should not be presented to the user. The second answer acquiring step acquires a second answer that the server has caused to be generated by a server-side language model. The answer outputting step outputs either the first answer or the second answer so that it can be presented to the user.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a reply program, a reply method, and a reply system. [Background technology]

[0002] Generative AI (Artificial Intelligence) services that use large-scale language models (LLMs) that generate answers in response to various user requests are becoming popular. A user can choose from multiple generative AI services. In addition, the user can select any one of multiple LLMs with different characteristics in the generative AI service and send a request. In this regard, a proposal has been disclosed for providing various services using multiple trained models.

[0003] For example, the information retrieval support device described in Patent Document 1 acquires question information, searches a first database, outputs first answer candidate information, acquires first evaluation information, and determines whether a second search is necessary based on the first evaluation information. If the information retrieval support device determines that a second search is necessary, it guides the user to the second search, searches a second database, outputs second answer candidate information, and acquires the second evaluation information. Furthermore, the information retrieval support device sets priority information based on the first evaluation information and the second evaluation information, and corrects at least one of the first answer candidate information and the second answer candidate information based on the priority information.

[0004] The method described in the patent document 2 also includes receiving first agent entry data, expecting a response within a first response domain, determining that the first user entry is not within the first response domain, identifying a second agent configured with a second response domain that includes the first user entry, and transmitting the first user entry to the second agent. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2024-34157 A [Patent Document 2] JP 2022-171624 A Summary of the Invention [Problem to be solved by the invention]

[0006] When applying the above-mentioned generative AI service using LLM, a large amount of computing resources is required compared to a local PC. Meanwhile, generative AI services using language models have been proposed for local environments where the scale of computing resources is relatively small, such as edge devices and local PCs. However, the generation results of language models used in local environments may be inferior in quality compared to LLMs used on servers. Therefore, quality assurance of generative AI services used in local environments is an issue.

[0007] In view of the above problems, the present disclosure aims to provide a technique for appropriately responding to a user request while making optimal use of computing resources. [Means for solving the problem]

[0008] A program according to the present disclosure causes an answer device having at least a first local language model and a second local language model to execute the following answer method. The answer method includes a request receiving step, a first answer generating step, a determination step, a request transmitting step, a second answer acquiring step, and an answer output step. The request receiving step receives a request including text data from a user. The first answer generating step causes the first local language model to generate a first answer based on the request. The determination step determines whether or not to present the first answer to the user by having a second local language model, which has a different execution environment from the first local language model, evaluate the first answer. The request transmitting step transmits a request to a server connected via a network when it is determined that the first answer is not to be presented to the user. The second answer acquiring step acquires a second answer generated by the server by supplying the request to the server-side language model. The answer output step outputs either the first answer or the second answer so as to be presentable to the user.

[0009] In the answering method according to the present disclosure, an answering device having at least a first local language model and a second local language model executes the following processes. The answering device receives a request including text data from a user. The answering device causes the first local language model to generate a first answer based on the request. The answering device determines whether to present the first answer to the user by causing a second local language model, which has an execution environment different from that of the first local language model, to evaluate the first answer. When the answering device determines not to present the first answer to the user, the answering device transmits a request to a server connected via a network. The answering device obtains a second answer generated by the server supplying the request to a server-side language model. The answering device presents either the first answer or the second answer to the user.

[0010] The answer system according to the present disclosure includes an answer device and a server. The answer device includes a first local language model, a second local language model, a request receiving unit, a first answer generating unit, a determination unit, a request transmitting unit, a second answer acquiring unit, and an answer output unit. The request receiving unit receives a request including text data from a user. The first answer generating unit causes the first local language model to generate a first answer based on the request. The determination unit determines whether or not to present the first answer to the user by having a second local language model, which has a different execution environment from the first local language model, evaluate the first answer. The request transmitting unit transmits a request to a server connected via a network when the determination unit does not determine that the first answer is to be presented to the user. The second answer acquiring unit acquires a second answer generated by the server by supplying a request to a server-side language model. The answer output unit and the answer device present either the first answer or the second answer to the user. The server is connected to the answer device described above via a network. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide a program, a reply method, and a reply system that appropriately reply to a user request while making optimal use of computing resources. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram of a response system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a hardware configuration of a computer. [Diagram 3] 4 is a flowchart of a reply method according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing a flow of information in the response system according to the first embodiment. [Diagram 5] FIG. 11 is a first diagram showing an image presented to a user who uses the answer system. [Figure 6] FIG. 11 is a block diagram of a reply system according to a second embodiment. [Figure 7] 13 is a flowchart of a reply method according to the second embodiment. [Figure 8] FIG. 11 is a first diagram showing a flow of information in a response system according to a second embodiment. [Figure 9] FIG. 11 is a second diagram showing the flow of information in the response system according to the second embodiment. [Figure 10] FIG. 2 is a second diagram showing an image presented to a user who uses the answer system. [Figure 11] FIG. 11 is a block diagram of a reply system according to a third embodiment. [Figure 12] FIG. 11 is a block diagram of a user terminal according to the third embodiment. [Figure 13] 13 is a flowchart of a reply method according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention will be described below through the embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. In addition, all of the configurations described in the embodiments are not necessarily essential as means for solving the problems. In order to clarify the explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary.

[0014] First Embodiment (Answer System 10) The answer system 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram of the answer system 10 according to the first embodiment. The answer system 10 accepts requests from users and presents answers to the accepted requests to the users. The answer system 10 mainly includes an answer device 100 and a server 200. The answer device 100 and the server 200 are connected to each other via a network N1 so as to be able to communicate with each other.

[0015] (Answer device 100) The answer device 100 is a computer, a tablet terminal, or a smartphone having a calculation function and a communication function. The answer device 100 has, as its main components, a first local language model 101, a second local language model 102, a request receiving unit 111, a first answer generating unit 112, a determining unit 113, a request sending unit 114, a second answer acquiring unit 115, and an answer output unit 116.

[0016] The first local language model 101 is a trained model trained to generate an answer by natural language processing for a request including text data. The first local language model 101 operates in the answering device 100. That is, the first local language model 101 generates an answer for a request received by the answering device 100 and outputs it to the answering device 100. The second local language model 102 is also a trained model trained to generate an answer for a predetermined request. Note that the architecture of the language model may have a neural network using deep learning, or may include a Generative Pre-trained Transformer (GPT). In this disclosure, the "language model" shown hereinafter is defined in the same manner as above. The language model may be referred to as a generative AI.

[0017] The request receiving unit 111 receives a request including text data from a user. More specifically, the request receiving unit 111 acquires text data or the like input by the user as a request.

[0018] The request may be data including image data, voice data, or other information in addition to text data. The data including other information may be, for example, a signal generated by a predetermined sensor. In this case, the first local language model 101 and the server-side language model 210 are multimodal language models. The multimodal language model may be called, for example, a multimodal large language model (MMLLM) or a multimodal model. The multimodal language model may generate a response by text data, or may generate a response including image data or voice data.

[0019] The first answer generation unit 112 causes the first local language model 101 to generate a first answer based on the request. For example, the first answer generation unit 112 may directly supply the request accepted by the request acceptance unit 111 to the first local language model 101 and cause the first local language model 101 to generate an answer. The first answer generation unit 112 may also perform a predetermined preprocessing on the request accepted by the request acceptance unit 111 and supply the preprocessed request to the first local language model 101. In this case, the predetermined preprocessing is, for example, a process of removing redundancy in a sentence or a process of correcting obvious typos.

[0020] The determination unit 113 determines whether or not to present the first answer to the user by having the second local language model 102, which has an execution environment different from that of the first local language model 101, evaluate the first answer. More specifically, the determination unit 113 asks the second local language model 102 whether or not the first answer has a generation quality equal to or higher than a predetermined level. In other words, the determination unit 113 causes the second local language model 102 to generate a result of determining whether or not the first answer has a generation quality equal to or higher than a predetermined level.

[0021] Generation quality may include, for example, factors such as accuracy, relevance, consistency, and customizability of the answer. Accuracy is a factor indicating whether the answer is factual. Relevance is a factor indicating whether the answer is directly related to the user's question. Coherence is a factor indicating whether the answer is logical and has a consistent content. Customizability is a factor indicating whether the answer is tailored to the user's individual needs and context.

[0022] In this case, the determination unit 113 inputs the first answer to be evaluated together with an instruction sentence instructing the second local language model 102 to evaluate the generation quality of the evaluation target. The determination unit 113 may also input a request. Then, as a response to this input, the determination unit 113 receives the result of evaluating the generation quality of the first answer from the second local language model 102. At this time, the evaluation result received by the determination unit 113 may be quantitative, qualitative, or a combination of these.

[0023] However, the evaluation result generated by the second local language model 102 is in a form that allows the determination unit 113 to make the above-mentioned determination. That is, when the generation quality of the first answer is equal to or higher than a predetermined standard, the determination unit 113 determines to present the first answer to the user. On the other hand, when the generation quality of the first answer is not equal to or higher than the predetermined standard, the determination unit 113 does not determine to present the first answer to the user.

[0024] Here, the "language models with different execution environments" include cases where the type of the language model itself is different. The "language models with different execution environments" include cases where the type of the language model is the same but the version is different. The "language models with different execution environments" include cases where the type and version of the language model are the same but the conditions instructed when generating an answer are different. The conditions instructed when generating an answer are, for example, the contents of the prompt instructions. In this case, the contents of the prompt instructions may include, for example, instruction policies such as "summarize", "suggest an idea", and "evaluate the text". Such instruction policies may also be referred to as "request types". In other words, when different prompts are input to language models of the same type and version, these can be said to be "language models with different execution environments". For example, after a request is input to one language model to generate a first answer, the language model is subsequently made to evaluate the first answer. In this case, the language model that generated the first answer may be referred to as the first local language model 101, and the language model that evaluated the first answer may be referred to as the second local language model 102, which has an execution environment different from that of the first local language model 101.

[0025] When the determination unit 113 does not determine that the first answer should be presented to the user, the request sending unit 114 sends a request to the server 200 connected via the network N1. In other words, the request sending unit 114 sends a request to the server 200 when the generation quality of the first answer is not equal to or higher than a predetermined standard. In other words, the request sending unit 114 refrains from sending a request to the server 200 when the generation quality of the first answer is equal to or higher than a predetermined standard.

[0026] The second answer acquisition unit 115 acquires the second answer generated by the server 200 by supplying a request to the server-side language model 210. That is, the second answer acquisition unit 115 acquires the second answer from the server 200 after the request sending unit 114 sends a request to the server 200.

[0027] The answer output unit 116 presents either the first answer or the second answer to the user. That is, the answer output unit 116 presents either the first answer or the second answer to the user depending on the result of the determination by the determination unit 113. More specifically, when the determination unit 113 determines that the first answer should be presented to the user, the answer output unit 116 presents the first answer to the user. On the other hand, when the determination unit 113 does not determine that the first answer should be presented to the user, the answer output unit 116 presents the second answer to the user.

[0028] In addition to the above-mentioned components, the response device 100 has a storage unit 121, a communication unit 122, an operation receiving unit 123, and a presentation unit .

[0029] The storage unit 121 stores various information to realize the functions of the response device 100 in the present disclosure. The storage unit 121 stores, for example, a program for executing the processing related to the present disclosure. The storage unit 121 may temporarily store a request received from a user, a first response generated by the first local language model 101, a second response acquired from the server 200, and the like.

[0030] The communication unit 122 has a function for the response device 100 to communicate with the server 200 via the network N1. For example, the communication unit 122 receives a user request from the request sending unit 114 and transmits the received request to the server 200. The communication unit 122 also supplies the second response received from the server 200 to the second response acquisition unit 115.

[0031] The operation receiving unit 123 receives an operation of a user who operates the response device 100. When the operation receiving unit 123 receives the user's operation, a request of the user is input to the response device 100. The input request is supplied to the request receiving unit 111. The operation receiving unit 123 also receives various operations other than input of a request.

[0032] The presentation unit 124 presents the first answer or the second answer to the user. The presentation unit 124 receives the first answer or the second answer from the answer output unit 116, and presents the received first answer or the second answer to the user. Specifically, the presentation unit 124 is, for example, a display unit that displays an image on a liquid crystal panel or an organic electroluminescence. In this case, the presentation unit 124 displays the content of the answer to the request in a manner that the user can visually recognize. The presentation unit 124 may also be configured to include, for example, a speaker. In this case, the presentation unit 124 outputs sound in a manner that the user can audibly recognize the content of the answer to the request.

[0033] (Server 200) The server 200 is a computer system communicatively connected to the answer device 100 via a network N1. The server 200 is connected to the above-mentioned answer device 100 via a network. The server 200 includes a server-side language model 210. The server 200 causes the server-side language model 210 to generate a second answer, which is an answer to a request received from the answer device 100, and transmits the second answer to the answer device 100 as a response to the request. The server-side language model 210 is, for example, an LLM. When the server 200 receives a request from the answer device 100, the server 200 supplies the request to the server-side language model 210 to generate an answer. The server 200 transmits the answer generated by the server-side language model 210 to the answer device 100.

[0034] The answering device 100 and the server 200 have been described above. In the answering system 10, the first local language model 101 has a smaller number of parameters than the server-side language model 210. The first local language model 101 may be a language model called a small language model (SLM). Similarly, the second local language model 102 may be an SLM.

[0035] If the first local language model 101 is an SLM and the server side language model 210 is an LLM, the following relationship may hold: the first local language model 101, being an SLM, has millions to billions of parameters, whereas the server side language model 210, being an LLM, has billions to trillions of parameters.

[0036] Furthermore, in the answer system 10, the first local language model 101 consumes less power when generating the first answer than the power consumption of the server-side language model 210 when generating the second answer. Furthermore, the processing time of the first local language model 101 when generating the first answer is longer than the processing time of the server-side language model 210 when generating the second answer. In the answer system 10, the first local language model 101 has fewer computing resources when generating the first answer than the computing resources of the server-side language model 210 when generating the second answer.

[0037] With the above-mentioned configuration, the answer system 10 prepares an answer to a request received from a user and presents the prepared answer to the user. The answer to the request is a first answer or a second answer. If the generation quality of the first answer generated by the first local language model 101 is equal to or higher than a predetermined standard, the answer system 10 presents the first answer to the user. On the other hand, if the generation quality of the first answer is not equal to or higher than the predetermined standard, the answer system 10 presents the second answer generated by the server-side language model 210 to the user. With the above-mentioned configuration, the answer system 10 ensures the generation quality of answers to user requests while making optimal use of computing resources.

[0038] (Hardware configuration example) Fig. 2 is a block diagram illustrating an example of a hardware configuration of a computer. The above-mentioned answer device 100 may have the configuration shown in Fig. 2. The computer 1000 has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0039] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0040] The processor 1020 is a circuit including an arithmetic device such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0041] The memory 1030 is a main storage device realized using a RAM (Random Access Memory) or the like.

[0042] The storage device 1040 is an auxiliary storage device such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or a read only memory (ROM), etc. The storage device 1040 stores programs for implementing the functions of the present disclosure.

[0043] The processor 1020 reads the program into the memory 1030 and executes it. This causes the processor 1020 to execute functions corresponding to the program. In other words, the program stored in the memory 1030 causes the computer 1000 to execute the functions of the present disclosure.

[0044] The input / output interface 1050 connects the computer 1000 to a predetermined input / output device. The input / output device is, for example, an input device such as a keyboard, an output device such as a display, or an input / output device in which a touch panel is superimposed on a display.

[0045] The network interface 1060 is an interface for connecting the computer 1000 to a predetermined communication network.

[0046] The computer 1000 has been described above, but in addition to the above-mentioned configuration, the computer 1000 may have an information input device that allows a user to input various information to the computer 1000 by operating it. The information input device is, for example, a keyboard, a mouse, a touch panel, etc. The computer 1000 may also have a display, a speaker, a vibration motor, an LED (light-emitting diode), etc., for showing various information to the user.

[0047] (How to answer) Next, a process executed by the response system 10 will be described with reference to Fig. 3. Fig. 3 is a flowchart of the response method according to the first embodiment. In the process executed by the response system 10 in the present disclosure, the response device 100 having at least a first local language model and a second local language model executes the following response method.

[0048] In step S11, the request receiving section 111 receives a request including text data from a user. The request receiving section 111 supplies the received request to the first answer generation section 112.

[0049] In step S12, the first answer generation unit 112 causes the first local language model 101 to generate a first answer based on the request. The first answer generation unit 112 supplies the first answer generated by the first local language model 101 to the determination unit 113.

[0050] In step S13, the determination unit 113 causes a second local language model, which has an execution environment different from that of the first local language model, to evaluate the first answer.

[0051] In step S14, the determination unit 113 determines whether or not to present the first answer to the user by having the second local language model evaluate the first answer. When the determination unit 113 determines that the first answer is to be presented to the user (step S14: YES), the answer device 100 proceeds to step S15. In this case, the determination unit 113 supplies the first answer to the answer output unit 116. On the other hand, when the determination unit 113 does not determine that the first answer is to be presented to the user (step S14: NO), the answer device 100 proceeds to step S16. In this case, the determination unit 113 supplies the request to the request transmission unit 114.

[0052] In step S15, the answer output unit 116 outputs the first answer received from the determination unit 113 so as to be presentable to the user. Specifically, the answer output unit 116 outputs the first answer to the presentation unit 124. The presentation unit 124 presents the received first answer to the user. Then, the answer system 10 ends a series of processes.

[0053] In step S16, if it is determined that the first answer should not be presented to the user, the request sending unit 114 sends a request to the server 200 connected via the network N1.

[0054] In step S17, the second answer acquisition unit 115 acquires the second answer generated by the server 200 by supplying the request to the server-side language model 210. The second answer acquisition unit 115 supplies the second answer acquired from the server 200 to the answer output unit 116.

[0055] In step S18, the answer output unit 116 outputs the second answer received from the second answer acquisition unit 115 so as to be presentable to the user. Specifically, the answer output unit 116 outputs the second answer to the presentation unit 124. The presentation unit 124 presents the received second answer to the user. Then, the answer system 10 ends a series of processes.

[0056] In the above-mentioned answering method, the first local language model 101 may be an SLM, and the server-side language model 210 may be an LLM. In this case, in step S12 of generating a first answer, the first answer generating unit 112 causes the first local language model 101 having a smaller number of parameters than the server-side language model 210 to generate the first answer. Then, if the generation quality of the first answer is not equal to or higher than a predetermined standard, in step S17, the server 200 causes the server-side language model 210 having a larger number of parameters than the first local language model 101 to generate a second answer. In this way, the answering system 10 suppresses consumption of computing resources while ensuring the generation quality of the answer.

[0057] In addition, in a generation AI service using the response system 10, the operator of this generation AI service may impose certain restrictions on the use of the server 200. The certain restrictions may be, for example, pay-per-use charges. Alternatively, the certain restrictions may be restrictions on the usage time or number of uses, etc. When such restrictions exist, the above-mentioned configuration allows the user to restrict the use of the server 200 and efficiently use the response system 10.

[0058] Specifically, for example, suppose that the operator of the generation AI service using the answer system 10 limits the number of times the server 200 can be used free of charge, and the user must pay a certain fee to remove the limit. In this case, the answer system 10 may have a counting unit, an access limiting unit, and a limit removal receiving unit. The counting unit counts the number of times the user has used the server 200 from the answer device 100. The access control unit limits the user's access to the server 200 when the number of times the server 200 has been used reaches a preset number. The limit removal receiving unit also receives the procedure for removing the limit by the user. The limit removal receiving unit may manage the payment process by the user. With this configuration, the answer system 10 can provide a mechanism that allows added value to be added to access to the server 200, which consumes relatively more computing resources than the answer device 100.

[0059] (Information flow) Next, the flow of information when the response system 10 executes the above-mentioned method will be described with reference to Fig. 4. Fig. 4 is a diagram showing the flow of information in the response system 10 according to the first embodiment.

[0060] First, the request receiving section 111 receives a request G11 from a user. The request receiving section 111 supplies the received request G11 to the first answer generation section 112.

[0061] Upon receiving the request G11, the first answer generation unit 112 causes the first local language model 101 to generate a first answer G12. The first answer generation unit 112 supplies the request G11 and the first answer G12 to the determination unit 113.

[0062] Upon receiving the request G11 and the first answer G12, the determination unit 113 inputs the received information into the second local language model 102 and causes the second local language model 102 to evaluate the first answer G12. The determination unit 113 determines, based on the result of the evaluation generated by the second local language model 102, whether or not to present the first answer G12 to the user.

[0063] When the determination unit 113 determines that the first answer should be presented to the user (step S14 in FIG. 3: YES), the determination unit 113 supplies the first answer G12 to the answer output unit 116. In this case, the answer output unit 116 outputs the first answer G12 received from the determination unit 113 to the presentation unit 124. On the other hand, when the determination unit 113 does not determine that the first answer should be presented to the user (step S14 in FIG. 3: NO), the determination unit 113 supplies the request G11 to the request transmission unit 114.

[0064] When the request sending unit 114 receives the request G11 from the determination unit 113, it sends the request G11 to the server 200. The server 200 supplies the request G11 to the server-side language model 210 to generate a second answer G21 that is an answer to the request G11. The server 200 sends the second answer G21 to the second answer acquisition unit 115. The second answer acquisition unit 115 supplies the received second answer G21 to the answer output unit 116. In this case, the answer output unit 116 outputs the second answer G21 received from the second answer acquisition unit 115 to the presentation unit 124.

[0065] The flow of information in the processes executed by the response system 10 has been described above. Each of the above processes may include information other than the information shown in the figure. Specifically, for example, each piece of information may be accompanied by a header and a footer, and the above information may be transferred as a payload. In addition, in this case, the header and footer may include information regarding the source of the information.

[0066] When the request sending unit 114 sends the request G11, it may also send information for identifying the request G11. In this case, the server 200 may send the information for identifying the request G11 and the second response G21 together to the second response acquisition unit 115. Alternatively, the server 200 may send the request G11 and the second response G21 to the second response acquisition unit 115. This allows the response device 100 to identify which request the second response G21 received from the server 200 is a response to.

[0067] (presentation information) Next, with reference to FIG. 5, information presented to a user by the response system 10 will be described. FIG. 5 is a first diagram showing an image presented to a user who uses the response system. FIG. 5 shows a display image 130. Here, it is assumed that the presentation unit 124 of the response device 100 has a display unit that displays the display image 130. It is also assumed that the response device 100 has an information input device in which a touch panel is superimposed on a screen that displays the display image 130. The display image 130 includes a request input field 131, a request display unit 132, and an answer display unit 133.

[0068] The request input field 131 is an information input field provided at the bottom of the display image 130. The request input field 131 has a text input field provided below a guide display saying "Please enter your request." When the user taps on the request input field 131, the display image 130 displays a virtual keyboard, making it possible to accept input of a request.

[0069] The request display section 132 displays a request input by a user. In Fig. 5, the request display section 132 displays a question text "What is a generation AI?" The answer system 10 generates an answer to the request displayed in the request display section 132.

[0070] The answer display unit 133 displays the answer output by the answer system 10 in response to the request displayed in the request display unit 132. In Fig. 5, the answer display unit 133 displays the answer text "Generative AI is a type of artificial intelligence...and is used in various fields."

[0071] This answer output by the answering system 10 does not indicate whether it was generated by the first local language model 101 or the server-side language model 210. In other words, the answering system 10 displays the answer on the answer display unit 133 without making the user aware of whether the answer was generated by the language model possessed by the answering device 100 or the language model possessed by the server 200. In other words, the user can recognize an answer with a certain generation quality guaranteed through the answering device 100 used by the user, regardless of whether the answer was generated by the first local language model 101 or the server-side language model 210.

[0072] The answer system 10 and the answer device 100 have been described above. Each of the components of the answer device 100 may be realized by dedicated hardware. In addition, a part or all of each component may be realized by a general-purpose or dedicated circuit, processor, or a combination of these. These may be configured by a single chip, or may be configured by multiple chips connected via a bus. A part or all of each component may be realized by a combination of the above-mentioned circuits and programs. In addition, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), or the like may be used as a processor. In addition, at least a part of the functions of this embodiment may be provided in the form of IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), or the like.

[0073] As described above, according to the present embodiment, it is possible to provide a program, a reply method, and a reply system that appropriately reply to a user request while making optimal use of computing resources.

[0074] <Second embodiment> Next, a second embodiment will be described. Fig. 6 is a block diagram of a response system 10 according to the second embodiment. In the response system 10 shown in Fig. 6, the configuration of the response device 100 is different from that of the response device 100 shown in Fig. 1.

[0075] The response device 100 according to this embodiment includes a third local language model 103 and a search function unit 104. The response device 100 also includes an intention information generation unit 117.

[0076] The third local language model 103 is a trained model trained to generate a response to a given request. When the third local language model 103 receives an instruction from the intention information generating unit 117 to generate intention information, it generates intention information as a response to the instruction. The intention information is information to be input to the first local language model 101 for the purpose of supplementing the request. The intention information supplements the intention of the user by generating information related to the user's request from the request input by the user. The intention information includes, for example, a request type in the request received from the user (an instruction policy such as "summarize", "suggest an idea", "translate", or "evaluate a sentence"). The prompt input to the first local language model 101 may include the request input by the user and the intention information.

[0077] When the third local language model 103 generates intention information, the search function unit 104 searches for words included in the request and for information related to those words, and supplies the search results to the first local language model 101. The search function unit 104 has a function called RAG (Retrieval-Augmented Generation). RAG is a mechanism for searching external information in text generation by a language model, and using the search results to generate an answer. RAG is also called "retrieval augmented generation" or "retrieval augmented generation." The first local language model 101 has the RAG function, so that the first local language model 101 can generate a first answer that is highly relevant to the request.

[0078] The intention information generation unit 117 causes the third local language model 103, which has an execution environment different from that of the first local language model 101, to generate intention information to be input to the first local language model 101 based on the request for the purpose of supplementing the request. In this way, the intention information generation unit 117 generates intention information including information regarding the type of the request from the context of the text data. The information regarding the type of the request is information including at least a request type.

[0079] The intention information generation unit 117 supplies the intention information generated by the third local language model 103 to the first answer generation unit 112. That is, in this case, the first answer generation unit 112 inputs at least a part of the intention information to the first local language model 101 to generate a first answer. If the first local language model 101 has a search function unit 104, the intention information may include content that instructs the first local language model 101 to search for information related to the request. In this case, the first local language model 101 searches for information included in the intention information in cooperation with the search function unit 104 based on the intention information.

[0080] As described above, the first answer generation unit 112 in this embodiment inputs at least a part of the intention information to the first local language model 101 to generate a first answer in the first local language model 101. This allows the answer device 100 to improve the quality of the answer generated by the first answer generation unit 112.

[0081] The determination unit 113 may receive the request, the first answer, and the intention information, output an evaluation score for the first answer to the second local language model 102, and perform the determination taking the evaluation score into consideration. This allows the answer system 10 to appropriately evaluate the generation quality of the first answer.

[0082] The determination unit 113 receives the request, the first answer, and the intention information as input, and outputs an evaluation score for the first answer generated by the first local language model 101 to the second local language model 102, which has an execution environment different from that of the first local language model 101. The evaluation score is an index indicating the quality of the generation. As described above, the generation quality may include elements such as the accuracy, relevance, consistency, and customizability of the answer. In other words, the second local language model 102 is configured to generate the evaluation score based on these elements.

[0083] The evaluation score is represented by an integer ranging from 0 to 100, for example. In this case, for example, a higher value may be defined as a higher generation quality. However, the evaluation score is not limited to the above definition. The evaluation score may be a quantitative one, or may be a qualitative indicator such as "good" or "bad."

[0084] (How to answer) Next, the process executed by the answering system 10 according to the present embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart of the answering method according to the second embodiment. The flowchart shown in Fig. 7 differs from the flowchart shown in Fig. 3 in that step S21 is provided between step S11 and step S12.

[0085] In step S11, the request receiving unit 111 receives a request from a user. The request receiving unit 111 supplies the received request to the intention information generating unit 117.

[0086] In step S21, the intention information generation unit 117 causes the third local language model 103 to generate intention information to be input to the first local language model 101 based on the request. Here, the third local language model 103 may generate intention information with content instructing to search for information related to the request. After generating the intention information, the intention information generation unit 117 supplies the generated intention information to the first answer generation unit 112.

[0087] (Information flow) Next, the flow of information accompanying the processing executed by the response system 10 according to the present embodiment will be described with reference to Fig. 8. Fig. 8 is a first diagram showing the flow of information in the response system according to the second embodiment.

[0088] First, upon receiving a request G11, the request receiving section 111 supplies the request G11 to the first answer generating section 112 and the intention information generating section 117.

[0089] When the intention information generation unit 117 receives the request G11, it causes the third local language model 103 to generate intention information G17. At this time, the third local language model 103 may generate content instructing the use of a search function. The intention information generation unit 117 supplies the intention information G17 generated by the third local language model 103 to the first answer generation unit 112.

[0090] When the first answer generation unit 112 receives the request G11 and the intention information G17, it supplies this information to the first local language model 101, causing the first local language model 101 to generate a first answer G12. At this time, if the intention information G17 includes content instructing the use of a search function, the first local language model 101 cooperates with the search function unit 104 to generate the first answer G12.

[0091] The flow of information explained in Fig. 8 will be continued with reference to Fig. 9. Fig. 9 is a second diagram showing the flow of information in the response system according to the second embodiment.

[0092] The first answer generation unit 112, which has caused the first local language model 101 to generate the first answer G12, supplies the determination unit 113 with the request G11, the first answer G12, and the intention information G17.

[0093] The determination unit 113 inputs the request G11, the first answer G12, and the intention information G17 to the second local language model 102, and causes the second local language model 102 to evaluate the generation quality of the first answer G12. Then, the determination unit 113 determines whether or not to present the first answer G12 to the user, depending on the evaluation result.

[0094] When the determination unit 113 determines that the first answer should be presented to the user (step S14 in FIG. 7: YES), the determination unit 113 supplies the first answer G12 to the answer output unit 116. In this case, the answer output unit 116 outputs the first answer G12 received from the determination unit 113 to the presentation unit 124. On the other hand, when the determination unit 113 does not determine that the first answer should be presented to the user (step S14 in FIG. 7: NO), the determination unit 113 supplies the request G11 to the request transmission unit 114. The process after the request transmission unit 114 receives the request G11 is the same as the information flow shown in FIG. 4. Therefore, a detailed description will be omitted here.

[0095] The flow of information in the second embodiment has been described above. The answer system 10 uses the intention information G17 in this way to improve the generation quality of the first answer G12 generated by the first local language model 101. Furthermore, when the generation quality of the first answer G12 is still not equal to or higher than a predetermined standard even after using the intention information G17, the answer system 10 causes the server side language model 210 of the server 200 to generate the second answer G21.

[0096] (presentation information) Next, information presented to a user by the response system 10 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a second diagram showing an image presented to a user who uses the response system 10.

[0097] The display image 130 shown in FIG. 10 differs from the display image 130 shown in FIG.

[0098] In the display image 130 shown in Fig. 10, the request display section 132 displays "How was the stock price yesterday?". In response, the answer display section 133 displays "Yesterday's ** average stock price was 39,154 yen. It was a drop of 1.1% from the previous day's closing price." Here, the "** average stock price" is a value indicating a representative index in securities trading. For example, a representative stock price index in Japan is the Nikkei Stock Average (registered trademark).

[0099] In this way, when a request from a user is a question such as "How was the stock price yesterday?", the answering system 10 causes the third local language model 103 to generate intention information related to this request. The intention information includes that "stock price" is related to the language used by the user. For example, when a user inputs a request in Japanese, the intention information generating unit 117 causes the third local language model 103 to generate intention information G17 including that the stock price is the Nikkei Stock Average (registered trademark), which is a representative stock price in the Japanese stock exchange, as information to supplement the "stock price". In addition, the first local language model 101 cannot generate the price of the stock price on the previous day. Therefore, the third local language model 103 reflects an instruction to search for stock price information using the search function unit 104 in the intention information. Therefore, the answering system 10 causes the first local language model 101 to generate the closing price of the ** average stock price on the previous day from the sentence "How was the stock price yesterday?".

[0100] The answer display section 133 also displays a reference information display section 134 and a language model display section 135. The reference information display section 134 displays the reference source of the information searched by the search function section 104 when the answer system 10 generates the answer shown in the answer display section 133. With such a display, the answer system 10 can present the basis of the answer to the user.

[0101] The language model display unit 135 is information about the language model that generated the answer displayed by the answer system 10 on the answer display unit 133. In the figure, "LM-003" is shown. In this case, "LM-003" is information indicating either the first local language model 101 or the server-side language model 210. By displaying the language model that generated the answer on the language model display unit 135, the answer system 10 can notify the user of the language model that generated the answer. In a generation AI service using the answer system 10, the operator of this generation AI service may impose certain restrictions on the user's use of the server 200. In this case, the user who uses the answer system 10 can recognize information about the language model used and grasp the usage status of the generation AI service.

[0102] Furthermore, in this embodiment, the answer system 10 displays the answer on the answer display unit 133 without allowing the user to select a language model included in the answer device 100. That is, the user can recognize an answer with a certain generation quality guaranteed through the answer device 100 used by the user, regardless of whether the answer is generated by the first local language model 101 or the server-side language model 210.

[0103] The second embodiment has been described above. In the answering system 10 according to this embodiment, the answering device 100 has the intention information generating unit 117, and thus generates an answer to the request G11 in a manner that suppresses the deterioration of the generation quality and also suppresses the consumption of computing resources. In addition, when the generation quality of the first answer G12 generated by the first local language model 101 is not determined to be equal to or higher than a predetermined standard, the server-side language model 210 is made to generate the second answer G21. In this way, the answering system 10 guarantees the generation quality. Therefore, according to this embodiment, it is possible to provide a program, an answering method, and an answering system that appropriately answers a user's request while favorably utilizing computing resources.

[0104] <Third embodiment> Next, a third embodiment will be described. Fig. 11 is a block diagram of an answering system 10 according to the third embodiment. In the answering system 10 shown in Fig. 11, a server 200 and a server-side language model 210 are connected via a network N1. In other words, the server 200 does not have the server-side language model 210. In this way, the server 200 may not have the server-side language model 210. In this case, the server 200 may have a predetermined API (Application Programming Interface) to exchange information with the server-side language model 210.

[0105] Moreover, the answer device 100 according to the present embodiment does not have a presenting unit 124. The answer device 100 according to the present embodiment is connected to a plurality of user terminals 300 (user terminal 300A, user terminal 300B, ...) via a local network N2. The answer device 100 according to the present embodiment is a local server in the local network N2. The local network N2 is, for example, a LAN (Local Area Network). In this disclosure, when the user terminal 300 is simply described, it is taken to collectively refer to the plurality of user terminals 300, unless otherwise specified.

[0106] The user terminals 300 are terminals used by individual users. The user terminals 300 are, for example, personal computers, tablet terminals, or smartphones. The user terminals 300 have a presentation unit that presents the first answer or the second answer to the user. The user terminals 300 are also connected to the response device 100 via a local network N2.

[0107] In the answer device 100, the request receiving unit 111 receives a request from one user terminal. In the answer device 100, when the determination unit 113 does not determine that the first answer is to be presented to the user, the request receiving unit 111 transmits the request to the server 200 via the network N1, which is a wide area network different from the local network N2.

[0108] The server 200 causes the server-side language model 210 to generate a second answer, which is an answer to the request received from the answer device 100. Then, the server 200 transmits the second answer to the answer device 100 as a response to the request.

[0109] The reply device 100 supplies the first reply or the second reply to one user terminal 300 that has accepted the request.

[0110] With the above-mentioned configuration, the answer device 100 accepts a request from any one of the user terminals 300. In this case, the answer device 100 outputs the first answer G12 or the second answer G21 in a presentable manner to one of the user terminals 300 that accepted the request. For example, assume that a user using the user terminal 300A transmits a request from the user terminal 300A. In this case, the answer device 100 outputs a response to this request to the user terminal 300A. Here, if the first answer generated in the first local language model 101 possessed by the answer device 100 can be presented to the user, the answer device 100 does not transmit a request to the server 200, but outputs the first answer to the user terminal 300A. On the other hand, if the first answer generated in the first local language model 101 cannot be presented to the user, the answer device 100 transmits a request to the server 200. Then, the answer device 100 causes the server-side language model 210, which consumes more computing resources than the first local language model 101, to generate a second answer, and outputs the second answer to the user terminal 300A. With this configuration, the answer device 100 can provide an answer system that generates a suitable answer while suppressing computing resources for a plurality of user terminals 300 connected to the local network N2.

[0111] (User terminal 300) The user terminal 300 will be described with reference to Fig. 12. Fig. 12 is a block diagram of the user terminal 300 according to the third embodiment. The user terminal 300 has, as its main components, a presentation unit 310, a storage unit 320, an operation reception unit 330, a communication unit 340, and a control unit 350.

[0112] The presentation unit 310 presents to the user the first answer or the second answer received from the response device 100. The presentation unit 310 is, for example, a display unit. The presentation unit 310 may be configured to include a speaker.

[0113] The storage unit 320 includes a non-volatile memory and stores, for example, an application program for implementing the functions of this embodiment. The operation reception unit 330 receives user requests by receiving operations of a keyboard, mouse, button, etc. The communication unit 340 has a function of communicatively connecting to the answer device 100 via the local network N2. The control unit 350 includes a calculation device such as a CPU, and controls each function of the user terminal 300.

[0114] (How to answer) Next, a method executed by the reply system 10 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart of the reply method according to the third embodiment. The flowchart shown in Fig. 13 differs from the flowchart shown in Fig. 7 in that it has step S31 instead of step S11. The flowchart shown in Fig. 13 also differs from the flowchart shown in Fig. 7 in that it has step S32 instead of step S15 and step S33 instead of step S18.

[0115] In step S31, the request receiving unit 111 receives a request from one user terminal 300. At this time, the request receiving unit 111 stores information about which user terminal 300 the request has been received from among the multiple user terminals 300. Specifically, the reply device 100 acquires identification information from the one user terminal 300 in communication with the user terminal 300. The identification information is, for example, an Internet Protocol (IP) address, a Media Access Control (MAC) address, or other identifiers that each user terminal 300 has. Furthermore, the request receiving unit 111 stores the identification information of the user terminal 300 that has received the request in association with the received request.

[0116] In step S32, the answer output unit 116 outputs the first answer generated by the first local language model 101 to the user terminal that has accepted the request. Here, the answer output unit 116 uses the information stored in step S31. That is, the answer output unit 116 refers to the identification information of the user terminal 300 linked to the request. In this way, the answer output unit 116 determines the output destination of the first answer to the request. The answer output unit 116 transmits the first answer to the determined one user terminal 300 via the local network N2.

[0117] In step S33, the answer output unit 116 outputs the second answer generated by the server-side language model 210 to the user terminal that has accepted the request. Here, the answer output unit 116 uses the information stored in step S31. That is, the answer output unit 116 refers to the identification information of the user terminal 300 linked to the request. In this way, the answer output unit 116 determines the output destination of the second answer to the request. The answer output unit 116 transmits the second answer to the determined one user terminal 300 via the local network N2.

[0118] The third embodiment has been described above. In this embodiment, the server 200 and the server-side language model 210 are configured separately, but the server 200 and the server-side language model 210 may be configured as an integrated unit as in the first embodiment.

[0119] According to the present embodiment, it is possible to provide a program, a reply method, and a reply system that appropriately reply to requests from a plurality of users while making optimal use of computing resources.

[0120] The above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other memory technology. The computer-readable medium or tangible storage medium includes a CD-ROM, a digital versatile disc (DVD), a Blu-ray® disk or other optical disk storage. The computer-readable medium or tangible storage medium includes a magnetic tape, a magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes an electrical, optical, acoustic, or other form of propagated signal.

[0121] Although the embodiments have been described above, the configurations of the above-described embodiments may be combined with each other, or some of the configurations may be replaced with other configurations. Furthermore, the configurations of the above-described embodiments may be modified in various ways without departing from the spirit of the invention.

[0122] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but to one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create an embodiment not explicitly shown or described, for example. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate. [Explanation of symbols]

[0123] 10 Answer System 100 answer device 101 First Local Language Model 102 Second Local Language Model 103 Third Local Language Model 104 Search Function Section 111 Request Reception Department 112 1st answer generation part 113 Judgment section 114 Request sending unit 115 2nd answer acquisition part 116 Answer output section 117 Intention Information Generation Unit 121 Storage section 122 Communications Department 123 Operation reception section 124 Presentation section 130 Display Images 131 Request input field 132 Request display section 133 Answer display section 134 Reference information display section 135 Language model display unit 200 Servers 210 Server-side language models 300 User terminals 310 Presentation section 320 Storage section 330 Operation reception section 340 Communications Department 350 Control section 1000 Computers 1010 Bus 1020 Processor 1030 Memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface G11 Request G12 1st answer G17 Intention Information G21 2nd answer N1 Network N2 Local Network

Claims

1. an answering device having at least a first local language model and a second local language model; a request receiving step of receiving a request including text data from a user; a first answer generating step of causing the first local language model to generate a first answer based on the request; a determining step of determining whether to present the first answer to the user by having the second local language model, the second local language model having an execution environment different from that of the first local language model, evaluate the first answer; a request sending step of sending the request to a server connected via a network when it is determined that the first answer should not be presented to the user; a second answer acquisition step of acquiring a second answer generated by the server by providing the request to a server-side language model; an answer output step of outputting either the first answer or the second answer so as to be presentable to the user; A method for answering comprising the steps of: Answer program.

2. 2. The reply program according to claim 1, the first answer generating step causes the first local language model, which has a smaller number of parameters than the server-side language model, to generate the first answer; Answer program.

3. 2. The reply program according to claim 1, an intention information generating step of generating, based on the request, intention information to be input to the first local language model in a third local language model having an execution environment different from that of the first local language model, for the purpose of supplementing the request; The first answer generating step includes inputting at least a part of the intention information into the first local language model to cause the first local language model to generate the first answer. Answer program.

4. 4. The reply program according to claim 3, the intention information generating step generates the intention information including information regarding the type of the request from a context of the text data. Answer program.

5. 4. The reply program according to claim 3, the intention information generating step causes the third local language model and a search function unit that searches for information within a predetermined network in cooperation with the third local language model to generate the intention information; Answer program.

6. In the reply program according to any one of claims 3 to 5, the determining step includes inputting the request, the first answer, and the intention information, causing the second local language model to output an evaluation score for the first answer, and performing the determination by taking the evaluation score into consideration. Answer program.

7. The reply program according to any one of claims 3 to 5, a counting step of counting the number of times the user has used the server from the answering device; an access control step of restricting access to the server by the user when the number of times the server has been used reaches a preset number; and a restriction release acceptance step of accepting a restriction release procedure by the user. Answer program.

8. an answering device having at least a first local language model and a second local language model; Accept a request including text data from a user, causing the first local language model to generate a first answer based on the request; determining whether to present the first answer to the user by having the second local language model, the second local language model having an execution environment different from that of the first local language model, evaluate the first answer; transmitting the request to a server connected via a network when it is determined that the first answer should not be presented to the user; the server providing the request to a server-side language model to generate a second answer; outputting either the first answer or the second answer so as to be presentable to the user; How to answer.

9. A first local language model; A second local language model; and a request receiving unit that receives a request including text data from a user; a first answer generation unit that causes the first local language model to generate a first answer based on the request; a determination unit that determines whether or not to present the first answer to the user by having the second local language model, the second local language model having an execution environment different from that of the first local language model, evaluate the first answer; a request sending unit that sends the request to a server connected via a network when it is determined that the first answer should not be presented to the user; a second answer acquisition unit that acquires a second answer generated by the server by providing the request to a server-side language model; a response device including a response output unit that presents either the first response or the second response to the user; The server is connected to the answering device via a network. Answer system.

10. 10. The response system according to claim 9, the first local language model has a smaller number of parameters than the server-side language model; Answer system.

11. 10. The response system according to claim 9, the first local language model consumes less power when generating the first answer than the power consumption of the server-side language model when generating the second answer, or a processing time when generating the first answer is longer than a processing time of the server-side language model when generating the second answer; Answer system.

12. In the reply system according to any one of claims 9 to 11, the server causes the server-side language model to generate the second answer, which is an answer to the request received from the answering device, and transmits the second answer to the answering device as a response to the request; the answering device further includes a presentation unit that presents the first answer or the second answer to the user. Answer system.

13. The reply system according to any one of claims 9 to 11, a user terminal having a presentation unit that presents the first answer or the second answer to the user and that is connected to the answer device via a local network; when the request receiving unit receives the request from one of the user terminals and the determination unit determines that the first answer should not be presented to the user, the request sending unit sends the request to the server via a wide area network different from the local network; the server causes the server-side language model to generate the second answer, which is an answer to the request received from the answering device, and transmits the second answer to the answering device as a response to the request; the answering device provides the first answer or the second answer to one of the user terminals that has accepted the request. Answer system.

Citation Information

Patent Citations

  • Information processing method and device, electronic equipment and storage medium

    CN117992592A

  • Data processing method and device, equipment, medium and program product

    CN118228015A

  • Computer-implemented method, computer program, and computer system for enhancing intent determination in messaging dialog manager system (enhancement of intent determination in messaging dialog manager system)

    JP2022171624A

  • Information retrieval support apparatus, information retrieval support method, program, and recording medium

    JP2024034157A

Cited By

  • Business support systems, business support methods, and programs

    JP2026136604A

  • Discussion support device, discussion support method, and program

    JP2026148028A

  • FAQ acquisition methods, FAQ acquisition systems, and FAQ acquisition programs

    JP2026168800A

  • Discussion support device, discussion support method, and program

    JP2026172568A

  • Information processing device, method, and program

    JP7764660B1