Information processing apparatus, information processing method, and information processing program

Abstract

To enable automatic adjustment so that a chatbot may generate consistent response information.SOLUTION: An information processing apparatus includes a determination unit and an adjustment unit. The determination unit determines whether or not first response information generated by a first chatbot satisfies a predetermined condition with respect to input information. The adjustment unit selects, when it is determined that the first response information does not satisfy the predetermined condition, a second chatbot having a language model different from that of the first chatbot, for generating response information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] In recent years, various voice assist functions using chatbots equipped with large language models have been proposed.

[0003] By the way, Patent Document 1 discloses a technique for requesting feedback from a user regarding one or more content parameters of a proposal or other content given by a chatbot.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the above-described conventional technology is a technology focused on the content of the proposal, such as improving future proposals or other content given to the user, and does not consider a mechanism for suppressing the non-uniformity of the response content generated by the chatbot.

[0006] Therefore, with the above-described conventional technology, it is not always possible to automatically adjust so that the chatbot can generate response information with consistent content.

[0007] The present invention has been made in view of the above, and proposes an information processing apparatus, an information processing method, and an information processing program that can automatically adjust so that the chatbot can generate response information with consistent content.

Means for Solving the Problems

[0008] The information processing apparatus according to claim 1 includes a determination unit that determines whether or not first response information generated by a first chatbot for input information satisfies a predetermined condition, and when the determination unit determines that the first response information does not satisfy the predetermined condition, an adjustment unit that selects, as an execution target for executing generation of response information, a second chatbot having a language model different from that of the first chatbot.

[0009] The information processing method according to claim 11 is an information processing method executed by an information processing apparatus, and includes a determination step of determining whether or not first response information generated by a first chatbot for input information satisfies a predetermined condition, and an adjustment step of selecting, as an execution target for executing generation of response information, a second chatbot having a language model different from that of the first chatbot when it is determined in the determination step that the first response information does not satisfy the predetermined condition.

[0010] The information processing program according to claim 12 is an information processing program executed by an information processing apparatus, and causes the information processing apparatus to execute a determination procedure of determining whether or not first response information generated by a first chatbot for input information satisfies a predetermined condition, and an adjustment procedure of selecting, as an execution target for executing generation of response information, a second chatbot having a language model different from that of the first chatbot when it is determined in the determination procedure that the first response information does not satisfy the predetermined condition.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] [Embodiment] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0013] One or more of the embodiments (including examples, modifications, and application examples) described below can each be implemented independently. On the other hand, at least a part of the plurality of embodiments described below may be implemented in appropriate combination with at least a part of other embodiments. These plurality of embodiments may include different novel features. Therefore, these plurality of embodiments can contribute to solving different purposes or problems and can exhibit different effects.

[0014] [1. Introduction] For example, it is possible to implement a service in which a terminal device obtains an answer corresponding to an input from a chatbot that communicates with a large language model via an API (Application Programming Interface) and presents it to the terminal device.

[0015] However, even if the same content instruction is input to the chatbot, responses with different meanings may be output. For example, when a user inputs "I want to go to Tokyo Tower.", the chatbot may respond with content along the lines of "The nearest station is Station A. It is accessible on foot from Station A.", which is in line with the user's intention of wanting route guidance. On the other hand, when the user inputs "I want to go to Tokyo Tower." again at a different timing, the response may be something like "Tokyo Tower is a representative tourist spot in Japan.", which has a different meaning from route guidance.

[0016] In this way, so-called response fluctuations (answer fluctuations) may occur, where the response content generated by the chatbot lacks consistency and becomes non-uniform. Note that response fluctuations can also be said to be unnaturalness in response generation.

[0017] Here, in some cases, a prompt with specific content is set to suppress response fluctuations. However, for example, when the large language model undergoes new learning and evolves or is updated, even if the same content prompt is set, new response fluctuations may occur. Then, the service provider is forced to modify the content of the prompt each time the response fluctuations, that is, responses with different meanings, increase.

[0018] The present invention proposes a mechanism that can improve the response accuracy of a chatbot by automatically adjusting the degradation when the consistency (appropriateness) of the response content of a large language model deteriorates.

[0019] In addition, when there is a limit to improving the response accuracy in automatic adjustment, the intervention of the service provider is required. However, as described above, it is cumbersome for the service provider to frequently perform correction work. Therefore, the present invention also proposes a mechanism for notifying that improvement of the response accuracy is impossible by automatic adjustment and that manual improvement is necessary at the timing when it becomes necessary.

[0020] [2. System Configuration] First, the configuration of the system according to the embodiment will be described with reference to FIG. 1. Hereinafter, the embodiments will be described separately in a plurality, but the system shown in FIG. 1 is common to all embodiments. FIG. 1 is a diagram showing an example of the system according to the embodiment. In FIG. 1, as an example of the system according to the embodiment, System 1 is shown. The information processing according to the proposed technology of the present invention is realized in System 1.

[0021] As shown in FIG. 1, System 1 includes a user device 10, an administrator device 30, an adjustment device 100, and a chatbot device 200. Further, the user device 10, the administrator device 30, the adjustment device 100, and the chatbot device 200 are communicably connected by wire or wirelessly via a network N. Also, in System 1, the number of each of the user device 10, the administrator device 30, the adjustment device 100, and the chatbot device 200 is not limited.

[0022] The user device 10 may be an information processing terminal used by a user U who intends to receive information provision through dialogue with a voice assistant. For example, the user device 10 may be a smartphone, a wearable device, a tablet-type terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like.

[0023] As another example, the user device 10 may be implemented as a navigation device built into or mounted on a vehicle, that is, an in-vehicle device. The user device 10 as an in-vehicle device may have not only a navigation function but also a recording function (a drive recorder function).

[0024] The administrator device 30 may be an information processing terminal used by an administrator T who manages the entire system 1 and the maintenance of the chatbot device 200. For example, the administrator device 30 may be a smartphone, a wearable device, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, or the like.

[0025] The adjustment device 100 is an example of an information processing device. As information processing according to the embodiment, the adjustment device 100 is an information processing device that performs adjustment processing for restoring consistency in the response content when a response fluctuation occurs in the chatbot, and notification processing for notifying that manual improvement is necessary when automatic improvement of the response accuracy by the adjustment processing is impossible. The functions of the adjustment device 100 may be realized by an information processing program according to the embodiment.

[0026] The chatbot device 200 realizes interaction with the user U as a voice assistant. The chatbot device 200 has a function of generating response information for input information by mounting a language model. For example, in the chatbot device 200, learning of the language model is repeated or the version of the language model is upgraded so as to realize high-precision interaction with the user U.

[0027] Note that the chatbot device 200 may be equipped with a so-called generative AI (artificial intelligence capable of creating various contents and ideas such as conversations, stories, images, videos, and music). Also, in the following embodiments, the "chatbot" shall substantially refer to the "chatbot device 200 (200A, 200B)".

[0028] In contrast to the user device 10 and the administrator device 30 being edge computers, the adjustment device 100 and the chatbot device 200 can be implemented as cloud computers.

[0029] Hereinafter, the embodiments will be described separately as a first embodiment and a second embodiment. However, when there is no need to distinguish between the first embodiment and the second embodiment, the information processing according to each embodiment will simply be referred to as "information processing according to the embodiment".

[0030] Also, the adjustment device 100 according to the first embodiment will be denoted as "adjustment device 100A", and the adjustment device 100 according to the second embodiment will be denoted as "adjustment device 100B". When there is no need to distinguish between "adjustment device 100A" and "adjustment device 100B", it will simply be denoted as "adjustment device 100".

[0031] Also, the chatbot device 200 according to the first embodiment will be denoted as "chatbot device 200A", and the chatbot device 200 according to the second embodiment will be denoted as "chatbot device 200B". When there is no need to distinguish between "chatbot device 200A" and "chatbot device 200B", it will simply be denoted as "chatbot device 200".

[0032] <First Embodiment> [1. Functional Configuration] Using FIG. 2, the configuration examples of the user device 10, the adjustment device 100A, and the chatbot device 200A according to the first embodiment will be described. FIG. 2 is a diagram showing an example of the device configuration according to the first embodiment. In FIG. 2, the administrator device 30 is omitted.

[0033] [User Device 10] As shown in FIG. 2, the user device 10 according to the first embodiment includes a communication unit 11, a storage unit 12, an input unit 13, an output unit 14, and a control unit 15.

[0034] (Communication Unit 11) The communication unit 11 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 11 is connected to the network N by wire or wirelessly, and performs information transmission and reception, for example, between the adjustment device 100 and the chatbot device 200.

[0035] (Memory unit 12) The memory unit 12 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, or a storage device such as a hard disk, an SSD (Solid State Drive), or an optical disk. The memory unit 12 may store various data related to the information processing according to the embodiment and input information.

[0036] (Input unit 13) The input unit 13 is an input device that receives various inputs from the outside. For example, the input unit 13 is an operating device for the user U to perform various operations, such as a keyboard, a mouse, and operation keys. When a touch panel is adopted in the user device 10, the touch panel is also included in the input unit 13. In this case, the user U performs various operations by touching the touch panel. The input unit 13 also includes a microphone that receives voice input by speech.

[0037] The user U may input various types of input information, such as "want to go to XX", "want to buy XX", "want to eat XX", "want to listen to XX", etc., via the input unit 13. The input information may be text or voice. The input information may be input to the chatbot device 200 via the adjustment device 100.

[0038] (Output unit 14) The output unit 14 is a device that performs various outputs to the outside, such as sound, light, vibration, and images. The output unit 14 performs various outputs to the user U according to the control of the control unit 15. Note that the output unit 14 may be a display device that displays various information. The display device is, for example, a liquid crystal display or an organic EL display (Organic Electro Luminescence Display). Note that the output unit 14 may be a touch panel type display device. In this case, the input unit 13 and the output unit 14 may be regarded as an integrated configuration. Also, the output unit 14 may be a speaker.

[0039] (Control unit 15) The control unit 15 is realized by various programs stored in the storage device inside the user device 10 being executed with the RAM as a working area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Also, the control unit 15 is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), for example.

[0040] As shown in FIG. 2, the control unit 15 has a transmission / reception unit 15a and an output control unit 15b, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in FIG. 2, and may be any other configuration as long as it performs the information processing described later. Also, the connection relationship of each processing unit included in the control unit 15 is not limited to the connection relationship shown in FIG. 2, and may be any other connection relationship.

[0041] (Transmission / reception unit 15a) The transmission / reception unit 15a receives the input information input via the input unit 13. For example, the transmission / reception unit 15a receives voice input by speech via a microphone or touch input via a touch panel. Then, the transmission / reception unit 15a transmits the received input information. For example, the transmission / reception unit 15a may transmit the received input information to the adjustment device 100, or may directly transmit the received input information to the chatbot device 200.

[0042] Also, the transmission / reception unit 15a receives the response information generated by the chatbot device 200 for the input information.

[0043] (Output control unit 15b) The output control unit 15b performs output control to cause the output unit 14 to output the response information generated by the chatbot device 200.

[0044] [Adjustment device 100A] As shown in FIG. 2, the adjustment device 100A according to the first embodiment includes a communication unit 110, a storage unit 120A, and a control unit 130A.

[0045] (Communication unit 110) The communication unit 110 is realized by, for example, a NIC or the like. The communication unit 110 is connected to the network N by wire or wirelessly, and performs information transmission and reception, for example, between the user device 10, the administrator device 30, and the chatbot device 200.

[0046] (Storage unit 120A) The storage unit 120A is realized by, for example, a semiconductor memory element such as a RAM, ROM, flash memory, or a storage device such as a hard disk, SSD, or optical disk. The storage unit 120A may store, for example, data and programs related to the information processing according to the first embodiment. Also, according to the example of FIG. 2, the storage unit 120A may include a correct answer information storage unit 121 and a prompt information storage unit 122.

[0047] (Correct answer information storage unit 121) The correct answer information storage unit 121 stores the correct response information (correct answer information) prepared for the input information. The correct response information may be prepared in advance by, for example, the administrator T.

[0048] (Prompt information storage unit 122) The prompt information storage unit 122 stores various prompts according to the first embodiment. Here, the prompt will be described. The prompt refers to an instruction sentence used for questions or instructions to the chatbot device 200 and corresponds to at least a part of the input information input to the chatbot device 200.

[0049] For example, the input information may be composed of a text that is the main idea of the instruction and a text that assists the instruction. In the following embodiments, the text that assists the instruction is defined as an "auxiliary prompt". A specific example will be shown in this regard. For example, assuming that the user U is a driver and the user U makes a speech input of "I want to go to the Tokyo Tower." to the input unit 13. In such a case, the adjustment device 100 performs intention interpretation based on the text "I want to go to the Tokyo Tower.", and the type of the input information according to the intention interpretation result (in this example, route guidance) is specified. Then, according to the adjustment device 100, an auxiliary prompt corresponding to the type of the input information is added.

[0050] As an example, "Please respond as the driver's assistant." may be provided as an auxiliary prompt. Here, the "driver's assistant" is a role determined by the adjustment device 100 that interprets the intention of the user U's speech input "I want to go to the Tokyo Tower." as a route guidance and decides that it should be requested from the chatbot device 200 in the context of route guidance. In such an example, "The driver wants to go to the Tokyo Tower. Please respond as the driver's assistant." is the input information, i.e., the prompt, and a part of it, "Please respond as the driver's assistant.", becomes the auxiliary prompt. On the other hand, the part other than the auxiliary prompt, specifically, "The driver wants to go to the Tokyo Tower.", may be the prompt, or the whole of "The driver wants to go to the Tokyo Tower. Please respond as the driver's assistant." may be the prompt.

[0051] Returning to the description of FIG. 1, the prompt information storage unit 122 may store such a candidate auxiliary prompt group used by the adjustment device 100 in this way. Also, as will be described later, the adjustment device 100 may dynamically generate input information in order to evaluate the response accuracy of the chatbot device 200. Therefore, the prompt information storage unit 122 may also store scenario information for the dynamic generation of input information (prompt).

[0052] Here, the information processing according to the first embodiment includes an adjustment process of adjusting the above-described auxiliary prompt according to the evaluation result of evaluating the response accuracy of the chatbot device 200 and repeating the adjustment of the auxiliary prompt until the evaluation result exceeds the pass line.

[0053] (Control Unit 130A) The control unit 130A is realized by various programs (for example, the information processing program according to the first embodiment) stored in the storage device inside the adjustment device 100A being executed with the RAM as the work area by a CPU, MPU, etc. Also, the control unit 130A is realized by an integrated circuit such as an ASIC or FPGA, for example.

[0054] As shown in FIG. 2, the control unit 130A includes an acquisition unit 131, a transmission unit 132, a determination information generation unit 133, a determination unit 134A, a prompt adjustment unit 135A, a notification unit 136A, a specifying unit 137, and a prompt providing unit 138, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130A is not limited to the configuration shown in FIG. 2, and any other configuration may be used as long as it can perform the information processing described later. Also, the connection relationship between the respective processing units included in the control unit 130A is not limited to the connection relationship shown in FIG. 2, and other connection relationships may be used.

[0055] (Acquisition Unit 131) The acquisition unit 131 acquires or receives various types of information in the information processing according to the embodiment. For example, the acquisition unit 131 acquires input information input to the chatbot device 200. For example, the acquisition unit 131 acquires, as input information input to the chatbot device 200, the input information received by the input unit 13. Also, the acquisition unit 131 may acquire the input information generated by the determination information generation unit 133.

[0056] The acquisition unit 131 acquires correct response information (correct answer information) prepared for the input information.

[0057] The acquisition unit 131 acquires the response information generated by the chatbot device 200 for the input information. Also, the acquisition unit 131 acquires the executed auxiliary prompt for which an adjustment process for adjusting the auxiliary prompt has been executed. Further, the acquisition unit 131 may acquire information of the language model.

[0058] (Transmission Unit 132) The transmission unit 132 transmits various types of information in the information processing according to the embodiment. For example, the transmission unit 132 transmits input information including an auxiliary prompt to the chatbot device 200. Also, the transmission unit 132 transmits the response information generated by the chatbot device 200 for the input information to the user device 10.

[0059] (Determination Information Generation Unit 133) The determination information generation unit 133 generates input information. For example, in the determination process of whether the response information generated by the chatbot device 200 satisfies a predetermined condition (that is, the evaluation process for evaluating the response accuracy of the chatbot device 200), input information for verifying the response accuracy is periodically input to the chatbot device 200. Therefore, the determination information generation unit 133 may generate this input information for verification, for example, at preset timings.

[0060] For example, the determination information generation unit 133 can generate input information in accordance with fixed scenarios such as "want to go to XX", "want to buy XX", "want to eat XX", "want to ask XX", etc. For example, when it is desired to evaluate the response accuracy of the chatbot device 200 for input information of the type of route guidance, the determination information generation unit 133 generates input information based on the fixed scenario "want to go to XX".

[0061] (Determination unit 134A) The determination unit 134A determines (evaluates) the response accuracy of the chatbot device 200A based on whether the response information generated by the chatbot device 200A in response to the input information satisfies a predetermined condition. Specifically, the determination unit 134A determines whether the first response information generated by the chatbot device 200A for the first input information satisfies a predetermined condition.

[0062] When it is determined that the first response information does not satisfy the predetermined condition and the response accuracy of the chatbot device 200A is low, an adjustment process for adjusting the auxiliary prompt included in the first input information is executed. Therefore, the determination unit 134A may further determine whether the second response information generated by the chatbot device 200A for the second input information including the auxiliary prompt for which the adjustment process has been executed satisfies a predetermined condition.

[0063] For example, each time the first input information with the same content is input, the determination unit 134A calculates the number of characters of each of the first response information generated by the chatbot device 200A. Then, based on the calculated number of characters, the determination unit 134A calculates a statistical value of the number of characters among a predetermined number of the first response information, and determines whether the statistical value satisfies a predetermined condition.

[0064] As another example, each time the first input information with the same content is input, the determination unit 134A calculates the similarity between each of the first response information generated by the chatbot device 200A and the correct answer information prepared in advance for the first input information. Then, based on the calculated similarity, the determination unit 134A calculates a statistical value of the similarity among a predetermined number of the first response information, and determines whether the statistical value satisfies a predetermined condition.

[0065] Note that the determination unit 134A may calculate quartiles, the mode, the minimum value, etc. as a plurality of types of statistical values, and determine whether more than a predetermined number of the plurality of types of statistical values satisfy a predetermined condition. For example, conditions corresponding to each statistical value such as quartiles, the mode, and the minimum value may be provided, and the determination unit 134A may evaluate the response accuracy based on whether a predetermined ratio or more of the plurality of types of statistical values satisfy the condition, or whether all of the plurality of types of statistical values satisfy the condition.

[0066] (Prompt adjustment unit 135A) When it is determined that the first response information does not satisfy a predetermined condition and the response accuracy of the chatbot device 200A is low, the prompt adjustment unit 135A adjusts the auxiliary prompt included in the first input information. Also, when it is determined that the second response information including the auxiliary prompt for which the adjustment process has been executed does not satisfy a predetermined condition and the response accuracy of the chatbot device 200A is still low, the prompt adjustment unit 135A adjusts the auxiliary prompt included in the second input information again.

[0067] In this way, the adjustment process of adjusting the auxiliary prompt is repeated until the response accuracy of the chatbot device 200A exceeds the passing line. Also, adjusting the auxiliary prompt corresponds to changing the auxiliary prompt. Based on these, the adjustment process of adjusting the auxiliary prompt corresponds to changing the auxiliary prompt to be adjusted at the current time to another auxiliary prompt.

[0068] Also, the process of changing the auxiliary prompt to be adjusted at the current time to another auxiliary prompt is classified into a process of changing the entire auxiliary prompt and a process of changing only some words included in the auxiliary prompt.

[0069] Specifically, the process of changing the entire auxiliary prompt will be described. The prompt adjustment unit 135A selects an arbitrary auxiliary prompt from a group of candidate auxiliary prompts prepared in advance according to the type of input information to the chatbot device 200A, and executes a process of swapping the auxiliary prompt to be adjusted at the current time and the selected auxiliary prompt. More specifically, when a predetermined condition is not satisfied, the prompt adjustment unit 135A selects an arbitrary auxiliary prompt from the group of candidate auxiliary prompts until the predetermined condition is satisfied, and repeatedly performs a swapping process of swapping from the auxiliary prompt to be adjusted at the current time to the selected auxiliary prompt.

[0070] Next, specifically, the process of changing only some words included in the auxiliary prompt will be described. The prompt adjustment unit 135A changes a predetermined word among the words included in the auxiliary prompt to be adjusted at the current time to another word, and swaps the auxiliary prompt to be adjusted at the current time and the changed auxiliary prompt after the predetermined word is changed to another word. More specifically, when a predetermined condition is not satisfied, the prompt adjustment unit 135A changes the predetermined word included in the auxiliary prompt to be adjusted at the current time to another word until the predetermined condition is satisfied, and repeatedly performs a swapping process of swapping from the auxiliary prompt to be adjusted at the current time to the changed auxiliary prompt after the predetermined word is changed to another word.

[0071] For example, the prompt adjustment unit 135A changes a word that designates a role corresponding to the type of input information to the chatbot device 200, as a predetermined word included in the auxiliary prompt to be adjusted at present, to a word that designates another role synonymous with the role.

[0072] (Notification unit 136A) According to the description so far, when it is determined that the response information generated by the chatbot device 200A according to the input information does not satisfy a predetermined condition and the response accuracy of the chatbot device 200A is low, an adjustment process for automatically improving the response accuracy is executed. Also, as described above, this adjustment process is repeated until the predetermined condition is satisfied, but there is a limit to the number of repetitions of the adjustment process. Therefore, when the predetermined condition is not satisfied even after reaching the limit and the automatic improvement of the response accuracy becomes impossible, the notification unit 136A notifies a predetermined notification destination that the automatic improvement of the response accuracy is impossible. For example, the notification unit 136A may notify the administrator device 30 of the administrator T of the information that improvement is impossible.

[0073] For example, when all the auxiliary prompts have been selected from the group of auxiliary prompts of the change candidates without the predetermined condition being satisfied, the notification unit 136A may notify the administrator T that the improvement of the response accuracy is impossible.

[0074] Also, when all of the other words prepared in a plurality have been used for the change without the predetermined condition being satisfied, the notification unit 136A may notify the administrator that the improvement of the response accuracy is impossible.

[0075] (Identification unit 137) The identification unit 137 identifies the type of the input information based on the result of the intentional interpretation of the input information. For example, when the input information of the user U is acquired, the identification unit 137 identifies the type of the input information (in this example, route guidance) according to the result of the intentional interpretation.

[0076] (Prompt provision unit 138) The prompt providing unit 138 provides an auxiliary prompt according to the type of input information to the input information. For example, the prompt providing unit 138 provides an auxiliary prompt to the input information of the user U.

[0077] [Chatbot device 200A] As shown in FIG. 2, the chatbot device 200A according to the first embodiment includes a communication unit 210, a storage unit 220A, and a control unit 230.

[0078] (Communication unit 210) The communication unit 210 is realized by, for example, a NIC or the like. And the communication unit 210 is connected to the network N by wire or wirelessly, and performs transmission and reception of information, for example, between the user device 10, the administrator device 30, and the chatbot device 200.

[0079] (Storage unit 220A) The storage unit 220A is realized by, for example, a semiconductor memory element such as a RAM, a ROM, a flash memory, or a storage device such as a hard disk, an SSD, or an optical disk. The storage unit 220A may store, for example, various data related to the information processing according to the first embodiment and the input information.

[0080] Also, the storage unit 220A may store a language model. FIG. 2 shows an example in which the storage unit 220A stores only one language model LLM1. Thus, in the first embodiment, the language model included in the chatbot device 200A may be fixed to one.

[0081] (Control unit 230) The control unit 230 is realized by various programs stored in a storage device inside the chatbot device 200A being executed with the RAM as a work area by a CPU, an MPU, or the like. Also, the control unit 230 is realized by an integrated circuit such as an ASIC or an FPGA.

[0082] As shown in FIG. 2, the control unit 230 includes a reception unit 231, a response information generation unit 232, and a transmission unit 233, and realizes or executes the functions and operations of information processing described below. Note that the internal configuration of the control unit 230 is not limited to the configuration shown in FIG. 2, and may be any other configuration as long as it can perform the information processing described later. Also, the connection relationship between the respective processing units included in the control unit 230 is not limited to the connection relationship shown in FIG. 2, and may be any other connection relationship.

[0083] (Reception unit 231) The reception unit 231 receives various types of information in the information processing according to the embodiment. For example, the reception unit 231 receives input information including an auxiliary prompt. Also, the reception unit 231 may receive the specification of a language model used for generating response information.

[0084] (Response information generation unit 232) When input information is received, the response information generation unit 232 generates response information for the received input information. For generating the response information, the specified language model among the language models stored in the storage unit 220A is used. For example, the response information generation unit 232 applies a language model to the input information and generates response information based on the output result by the language model. From this, it can be said that the response information generation unit 232 is a processing unit that substantially realizes the function as a chatbot.

[0085] (Transmission unit 233) The transmission unit 233 transmits the response information generated by the response information generation unit 232. When the input information is received from the user U, the transmission unit 233 transmits the response information generated for this input information to the user device 10. As a result, a dialogue is established between the user and the chatbot. On the other hand, when the input information is verification input information input by the adjustment device 100, the transmission unit 233 transmits the response information generated for this input information to the adjustment device 100.

[0086] [2. Overall processing procedure of the information processing according to the first embodiment] FIG. 3 is a diagram showing the overall flow of an information processing procedure according to a first embodiment realized by the adjustment device 100A. The information processing according to the first embodiment includes an evaluation process for evaluating the response accuracy of the chatbot device 200A and an adjustment process for adjusting the auxiliary prompt. And these processes are repeated until the response accuracy of the chatbot device 200A satisfies the conditions. Therefore, first, the flow of the first round of the information processing according to the first embodiment will be described, and then the flow of the second round and subsequent rounds of the information processing according to the first embodiment will be described.

[0087] Also, in the example of FIG. 3, a scene where the auxiliary prompt is adjusted during the verification experiment for evaluating the response accuracy of the chatbot device 200A is shown. In such a verification experiment scenario, the verification input information may be generated by the determination information generation unit 133 at specific timings (for example, once a day).

[0088] On the other hand, the evaluation of the response accuracy of the chatbot device 200A and the adjustment of the auxiliary prompt may be performed in the actual scenario where the chatbot device 200A is utilized by the user U. In such an example, instead of the verification input information generated by the determination information generation unit 133, the input information actually input by the user U in the utilization scenario may be used for the evaluation process and the adjustment process.

[0089] Also, there are various types of input information that the chatbot device 200A can handle, such as "route guidance", "cooking recipes", "music content", etc. In FIG. 3, a scene where the automatic adjustment of the chatbot device 200A in the field of "route guidance" is shown.

[0090] (First round of processing) The determination information generation unit 133 determines whether it is the timing (for example, the timing when it is 16:00 once a day) to execute the evaluation process for evaluating the response accuracy of the chatbot device 200A (step S101). If the determination information generation unit 133 is not at the timing to execute the evaluation process (step S101; No), it waits until it is the timing to execute the evaluation process.

[0091] On the other hand, when it is time to execute the evaluation process (step S101; Yes), the determination information generation unit 133 generates input information for verification based on a fixed scenario (step S102). In the example of FIG. 3 where the automatic adjustment of the chatbot device 200A in the "route guidance" field is performed, the determination information generation unit 133 is assumed to generate input information such as "I want to go to the Tokyo Tower." using the fixed scenario "want to go to XX". Note that the determination information generation unit 133 may generate input information with the same content each time.

[0092] Next, the prompt addition unit 138 determines whether the adjustment process has not been executed at the current time (step S103). In the first round of processing, the prompt addition unit 138 determines that the adjustment process has not been executed at the current time (step S103; not executed), and initializes (initially adds) an auxiliary prompt to the input information generated in step S104 (step S104). For example, the prompt addition unit 138 may select any one auxiliary prompt from the auxiliary prompt group of change candidates stored in the prompt information storage unit 122. The auxiliary prompts for change candidates may be prepared in advance for each type of input information that the chatbot device 200A can handle. Therefore, in the example of FIG. 3, the prompt addition unit 138 may select any one auxiliary prompt from the auxiliary prompts for change candidates corresponding to "route guidance". For example, the prompt addition unit 138 selects the auxiliary prompt PR1 "Please respond as a driver's assistant." and assigns it to the input information generated in step S102.

[0093] The transmission unit 132 transmits the input information IN1 including the auxiliary prompt PR1 to the chatbot device 200A (step S105). The chatbot device 200A applies the language model LLM1 to the input information IN1 and generates response information AN1 based on the output result of the language model LLM1.

[0094] Note that the input information may be the same each time it is generated. However, even if the input information of the same content is input, the chatbot device 200A does not necessarily always generate the response information AN1 of the same content. For example, the chatbot device 200A may generate response information in line with the intention of wanting route guidance, or may generate response information AN1 with a different purport from route guidance. That is, there may be fluctuations in the content of the response information.

[0095] The acquisition unit 131 acquires the response information AN1 generated by the chatbot device 200A for the input information IN1 (step S106).

[0096] The determination unit 134A determines whether or not a predetermined number of response information AN1 generated by the chatbot device 200A for the input information IN1 has been accumulated (for example, 100 have been accumulated) (step S107). If it is determined by the determination unit 134A that the response information AN1 has not been accumulated by the predetermined number (step S107; No), the process returns to step S102 and is repeated until the response information AN1 is accumulated by the predetermined number.

[0097] On the other hand, when the determination unit 134A determines that the response information AN1 has been accumulated by the predetermined number (step S107; Yes), it executes an evaluation value calculation process for response accuracy using the predetermined number of response information AN1 (step S108). The detailed procedure of the evaluation process performed in step S108 will be described with reference to FIGS. 4 and 5. Here, although each of the response information AN1 accumulated by the predetermined number is a response to the input information IN1 of the same content, due to the response fluctuations of the chatbot device 200A, there may be responses of different contents. Therefore, the determination unit 134A evaluates the response accuracy of the chatbot device 200A based on whether or not the evaluation value calculated in step S108 satisfies a predetermined condition (step S109).

[0098] When the evaluation value of the response accuracy satisfies a predetermined condition (step S109; Yes), the prompt adjustment unit 135A registers the current auxiliary prompt (for example, the auxiliary prompt PR1) as a generation algorithm capable of realizing the chatbot device 200A with high response accuracy (step S110). That is, the prompt adjustment unit 135A registers the auxiliary prompt whose response accuracy has been verified as the auxiliary prompt to be given by the prompt giving unit 138 to the input information of the user U. Then, the process ends.

[0099] On the other hand, when it is determined that the evaluation value of the response accuracy does not satisfy the predetermined condition (step S109; No), the prompt adjustment unit 135A executes an adjustment process for adjusting the auxiliary prompt (step S113). The detailed procedure of the adjustment process performed in step S113 will be described with reference to FIGS. 6 and 7.

[0100] After the adjustment process is performed by the prompt adjustment unit 135A, the process returns to step S102 and proceeds to the second round of processing.

[0101] (After the second round of processing) In and after the second round of processing, the determination information generation unit 133 generates verification input information based on a fixed scenario (step S102). The determination information generation unit 133 may generate the same input information "I want to go to the Tokyo Tower." as in the first round of processing for the second and subsequent rounds of processing.

[0102] Next, the prompt giving unit 138 determines whether or not the adjustment process has not been executed at the current time (step S103). In the case of the second and subsequent rounds of processing, the prompt giving unit 138 determines that the adjustment process has not been executed at the current time, that is, it has been executed (step S103; executed), acquires the auxiliary prompt PR2 for which the adjustment process has been executed in step S113, and gives the acquired auxiliary prompt PR2 to the input information generated in step S102 (step S114).

[0103] The transmission unit 132 transmits the input information IN2 including the auxiliary prompt PR2 to the chatbot device 200A (step S115). The chatbot device 200A applies the language model LLM1 to the input information IN2 and generates response information AN2 based on the output result of the language model LLM1.

[0104] The acquisition unit 131 acquires the response information AN2 generated by the chatbot device 200A for the input information IN2 (step S116).

[0105] The determination unit 134A determines whether or not a predetermined number of response information AN2 generated by the chatbot device 200A for the input information IN2 has been accumulated (for example, 100 have been accumulated) (step S107). If the determination unit 134A determines that the response information AN2 has not been accumulated by a predetermined number (step S107; No), the process returns to step S102 and is repeated until the response information AN2 is accumulated by a predetermined number.

[0106] On the other hand, when the determination unit 134A determines that the response information AN2 has been accumulated by a predetermined number (step S107; Yes), it executes an evaluation value calculation process for the response accuracy using the predetermined number of response information AN2 (step S108). Here, although each of the response information AN2 accumulated by a predetermined number is a response to the input information IN2 of the same content, due to the response variation of the chatbot device 200A, there may be responses of different contents. Therefore, the determination unit 134A evaluates the response accuracy of the chatbot device 200A based on whether or not the evaluation value calculated in step S108 satisfies a predetermined condition (step S109).

[0107] When the evaluation value of the response accuracy satisfies a predetermined condition (step S109; Yes), the prompt adjustment unit 135A registers the current auxiliary prompt (for example, the auxiliary prompt PR2) as a generation algorithm capable of realizing the chatbot device 200A with high response accuracy (step S110). Here, in the past process, the current auxiliary prompt (for example, the auxiliary prompt PR2) is newly registered by replacing it with the auxiliary prompt (for example, the auxiliary prompt PR1) registered as a generation algorithm capable of realizing the chatbot device 200A with high response accuracy. That is, the prompt adjustment unit 135A registers the auxiliary prompt verified to have high response accuracy as an adjusted auxiliary prompt to be given to the input information of the user U by the prompt giving unit 138. Then, the process ends.

[0108] On the other hand, even at the current time after the second round of processing, there may be a case where no auxiliary prompt that can obtain an evaluation value satisfying the predetermined condition is found. When it is determined that the evaluation value of the response accuracy does not satisfy the predetermined condition (step S109; No), the prompt adjustment unit 135A executes the adjustment process for adjusting the auxiliary prompt again (step S113).

[0109] After the adjustment process is performed by the prompt adjustment unit 135A, the process returns to step S102 and proceeds to the next round.

[0110] 〔3. Specific Procedure of Evaluation Process〕 Subsequently, the specific procedure of the evaluation process performed in step S108 of FIG. 3 will be described. The evaluation process includes a pattern using the number of characters of the response information generated by the chatbot device 200A and a pattern using the similarity between the response information generated by the chatbot device 200A and the correct response information. In FIG. 4, the former processing procedure a is described, and in FIG. 5, the latter processing procedure b is described.

[0111] FIG. 4 is a diagram showing the specific procedure (1) of the evaluation process according to the embodiment. First, the acquisition unit 131 acquires all of the response information accumulated by a predetermined number (for example, 100 pieces are accumulated) (step S1081a). The acquired response information includes the response information AN1 and the response information AN2 shown in FIG. 3.

[0112] The determination unit 134A calculates the number of characters in each of the response information (step S1082a). Further, the determination unit 134A calculates the statistical value of the number of characters among the response information as an evaluation value for evaluating the response accuracy of the chatbot device 200A (step S1083a). For example, it can be said that the more the response information generated by the chatbot device 200A is redundant, the more it contains unnecessary information that deviates from the user's intention. Therefore, by focusing on the number of characters, the response accuracy of the chatbot device 200A can be appropriately evaluated.

[0113] Note that the determination unit 134A may calculate at least one of, for example, the quartile, the mode, or the minimum value as the statistical value.

[0114] Then, as the evaluation process of step S109 in FIG. 2 for evaluating the response accuracy of the chatbot device 200A based on whether the evaluation value satisfies a predetermined condition, the determination unit 134A may perform a process of determining whether the statistical value of the number of characters calculated in S1083a satisfies the threshold condition.

[0115] Here, the fact that the statistical value of the number of characters satisfies the threshold condition (step S109; Yes) means that the response accuracy of the chatbot device 200A is high, so the process proceeds to step S110.

[0116] On the other hand, the fact that the statistical value of the number of characters does not satisfy the threshold condition (step S109; No) means that the response accuracy of the chatbot device 200A is low, so the process proceeds to step S113.

[0117] Note that when the determination unit 134A calculates, for example, the quartiles, the mode, and the minimum value as multiple types of character count statistics, it may perform conditional determination individually, such as whether the quartiles satisfy the threshold condition, whether the mode satisfies the threshold condition, and whether the minimum value satisfies the threshold condition. Then, the determination unit 134A may evaluate the response accuracy based on whether each of the statistics of a predetermined ratio or more (for example, the majority) of the multiple types of character count statistics (for example, the quartiles and the mode) satisfies the threshold condition, or whether each of all the statistics of the multiple types of character counts (the quartiles, the mode, and the minimum value) satisfies the threshold condition.

[0118] Next, the process of FIG. 5 will be described. FIG. 5 is a diagram showing a specific procedure (2) of the evaluation process according to the embodiment. Also in the example of FIG. 5, the acquisition unit 131 acquires all of the response information accumulated by a predetermined number (for example, 100 pieces are accumulated) (step S1081b). The acquired response information includes the response information AN1 and the response information AN2 shown in FIG. 3.

[0119] Also, the acquisition unit 131 acquires correct answer information (correct response information) prepared in advance for the input information (step S1082b).

[0120] Then, the determination unit 134A calculates the similarity between each of the response information and the correct answer information (step S1083b). Also, the determination unit 134A calculates a statistical value of the similarity between the response information as an evaluation value for evaluating the response accuracy of the chatbot device 200A (step S1084a). For example, the determination unit 134A may calculate the cosine similarity between each of the response information and the correct answer information. For example, it can be said that the more similar the response information generated by the chatbot device 200A is to the correct answer information, the more the response information is composed of only appropriate information suitable for the user's intention. Therefore, by focusing on the similarity, the response accuracy of the chatbot device 200A can be appropriately evaluated.

[0121] Then, based on whether the evaluation value satisfies a predetermined condition, the determination unit 134A may perform a process of determining whether the statistical value of the similarity calculated in S1083a satisfies a threshold condition as the evaluation process in step S109 of evaluating the response accuracy of the chatbot device 200A in FIG. 2.

[0122] Here, that the statistical value of the similarity satisfies the threshold condition (step S109; Yes) means that the response accuracy of the chatbot device 200A is high, so the process proceeds to step S110.

[0123] On the other hand, that the statistical value of the similarity does not satisfy the threshold condition (step S109; No) means that the response accuracy of the chatbot device 200A is low, so the process proceeds to step S113.

[0124] Note that when the determination unit 134A calculates, as a plurality of types of similarity statistical values, for example, the quartile, the mode, and the minimum value, the condition determination may be performed individually such as whether the quartile satisfies the threshold condition, whether the mode satisfies the threshold condition, and whether the minimum value satisfies the threshold condition. Then, the determination unit 134A may evaluate the response accuracy based on whether each of the statistical values (for example, the quartile and the mode) of a predetermined ratio or more (for example, the majority) among the plurality of types of similarity statistical values satisfies the threshold condition, or whether each of all the statistical values (the quartile, the mode, and the minimum value) among the plurality of types of similarity statistical values satisfies the threshold condition.

[0125] Note that FIGS. 3 and 4 show the evaluation processing procedure as a method for evaluating the response accuracy of the chatbot device 200A according to the first embodiment. However, the response accuracy of the chatbot device 200B according to the second embodiment is also evaluated by a similar method. Therefore, in the second embodiment described later, the description of the specific procedure of the evaluation process for the chatbot device 200B is omitted.

[0126] 〔4. Specific Procedure of Adjustment Processing〕 Next, the specific procedure of the adjustment process performed in step S113 of FIG. 3 will be described. The adjustment process for adjusting the auxiliary prompt is classified into a process of changing the entire auxiliary prompt and a process of changing only some words included in the auxiliary prompt. In FIG. 6, the former processing procedure a will be described, and in FIG. 7, the latter processing procedure b will be described.

[0127] FIG. 6 is a diagram showing the specific procedure (1) of the adjustment process according to the first embodiment. The prompt adjustment unit 135A determines whether there is an unselected auxiliary prompt among the auxiliary prompt groups prepared in advance (step S1131a). For example, the prompt adjustment unit 135A designates an auxiliary prompt group of change candidates corresponding to the type "route guidance" among the auxiliary prompt groups of change candidates prepared in advance for each type of input information that the chatbot device 200A can handle. Then, the prompt adjustment unit 135A may determine whether there is an unselected auxiliary prompt among the auxiliary prompt groups of change candidates corresponding to the type "route guidance".

[0128] If there is an unselected auxiliary prompt among the auxiliary prompt groups prepared in advance (step S1131a; Yes), the prompt adjustment unit 135A selects one of the unselected auxiliary prompts (step S1132a). For example, assume that the auxiliary prompt PR1 "Please respond as a driver's assistant." is the adjustment target at the current time, and there are auxiliary prompts PRC11 and PRC12 as shown in FIG. 6 as unselected auxiliary prompts. In such an example, the prompt adjustment unit 135A can select either one of the auxiliary prompts PRC11 and PRC12.

[0129] Then, the prompt adjustment unit 135A replaces the auxiliary prompt that is the adjustment target at the current time with the auxiliary prompt selected in step S1132a (step S1133a).

[0130] Subsequently, the prompt adjustment unit 135A holds the auxiliary prompt after replacement as the auxiliary prompt for which the adjustment process has been executed (step S1134a). Then, the process proceeds to step S101. For example, assume that the auxiliary prompt PR1 such as "Please respond as a driver's assistant." is replaced with the auxiliary prompt "Please guide a destination according to the driver's preference." listed as a candidate. In such an example, the auxiliary prompt "Please guide a destination according to the driver's preference." is held as the auxiliary prompt PR2 for which the adjustment process has been executed and will be acquired in step S114 of FIG. 3.

[0131] On the other hand, when there is no unselected auxiliary prompt in the group of auxiliary prompts prepared in advance (step S1131a; No), that is, when all the auxiliary prompts prepared in advance have been used up, the prompt adjustment unit 135A recognizes that no matter which auxiliary prompt is given to the chatbot device 200A, response information cannot be generated so as to obtain an evaluation value that satisfies the conditions (the response accuracy of the chatbot device 200A has not been improved no matter which auxiliary prompt is used) (step S1135a).

[0132] Then, the notification unit 136A notifies the administrator T that it is impossible to improve the response accuracy of the chatbot device 200A (step 1136a).

[0133] Next, the process of FIG. 7 will be described. FIG. 7 is a diagram showing the specific procedure (2) of the adjustment process according to the embodiment. The prompt adjustment unit 135A extracts role-designating words from among the words included in the auxiliary prompt to be adjusted at the current time (step 1131b). The role-designating word refers to a word that designates a role according to the type of input information that the chatbot device 200A can handle. For example, as role-designating words corresponding to the type "route guidance", there are "assistant", "butler", "passenger", "driving partner", etc. And "assistant", "butler", "passenger", "driving partner" have a relationship of synonymy with each other although their expressions are different.

[0134] The prompt adjustment unit 135A determines whether there is an unselected role designating word among a group of role designating words prepared in advance (step S1132b).

[0135] If there is an unselected role designating word among the group of role designating words prepared in advance (step S1132b; Yes), the prompt adjustment unit 135A selects one of the unselected role designating words (step S1133b). For example, the auxiliary prompt PR1 such as "Please respond as the driver's assistant." is the adjustment target at the current time, and assume that there are role designating words WDC21, WDC22, and WDC23 as shown in FIG. 7 as unselected role designating words. In such an example, the prompt adjustment unit 135A can select any one of the role designating words WDC21 to WDC23.

[0136] Note that the role designating words may be prepared in advance by, for example, the administrator T and registered in the prompt information storage unit 122. On the other hand, the role designating words may be, for example, those listed by the chatbot device 200. For example, the administrator T may give an instruction such as "Please think of words representing the role of assisting the driver" to request the chatbot device 200 to present role words.

[0137] Returning to the description of FIG. 7, the prompt adjustment unit 135A changes the role designating word included in the auxiliary prompt that is the adjustment target at the current time to the role designating word selected in step S1133b (step S1134b).

[0138] Also, the prompt adjustment unit 135A swaps the auxiliary prompt that is the adjustment target at the current time and the auxiliary prompt after the role designating word is changed (step S1135b).

[0139] Subsequently, the prompt adjustment unit 135A holds the auxiliary prompt after replacement as the auxiliary prompt for which the adjustment process has been executed (step S1136b). Then, the process proceeds to step S101. For example, assume that the role designation word "assistant" included in the auxiliary prompt PR1 "Please respond as a driver's assistant." is changed to another role designation word "butler". In such an example, the auxiliary prompt "Please respond as the driver's butler." is held as the auxiliary prompt PR2 for which the adjustment process has been executed and will be obtained in step S114 of FIG. 3.

[0140] On the other hand, when there is no unselected role designation word among the group of role designation words prepared in advance (step S1132b; No), that is, when all the role designation words prepared in advance have been used up, no response information can be generated so that an evaluation value satisfying the conditions can be obtained no matter which auxiliary prompt including a role designation word is given to the chatbot device 200A (the response accuracy of the chatbot device 200A has not been improved no matter which auxiliary prompt including a role designation word is used). It is recognized (step S1137b).

[0141] Then, the notification unit 136A notifies the administrator T that it is impossible to improve the response accuracy of the chatbot device 200A (step 1138b).

[0142] [5. Dialogue control process using the adjusted auxiliary prompt] Assume that the adjustment device 100A according to the first embodiment can adjust an auxiliary prompt capable of realizing a chatbot device 200A with high response accuracy by the adjustment process described so far. In such a case, the adjustment device 100A controls so that the chatbot device 200A can generate response information corresponding to the input information of the user U according to the adjusted auxiliary prompt. Such dialogue control will be described with reference to FIG. 8. FIG. 8 is a diagram showing the procedure of dialogue control executed by the adjustment device 100A according to the first embodiment.

[0143] First, the acquisition unit 131 determines whether it has acquired the input information INU of the user U (step S201).

[0144] Based on the result of the intentional interpretation of the input information INU, the specifying unit 137 specifies the type of the input information INU (step S202).

[0145] The prompt providing unit 138 acquires an auxiliary prompt corresponding to the type of the input information INU from among the auxiliary prompts (that is, the auxiliary prompts that can realize the chatbot device 200A with high response accuracy) that can obtain an evaluation value satisfying the conditions (step S203).

[0146] Then, the prompt providing unit 138 attaches the auxiliary prompt acquired in step S203 to the input information INU (step S204). For example, among the types of input information that the chatbot device 200A can handle, assume that the type of the input information INU is "route guidance" and the auxiliary prompt PR2 is registered as an auxiliary prompt that can realize the chatbot device 200A with high response accuracy. In such an example, the prompt providing unit 138 acquires the auxiliary prompt PR2 and attaches it to the input information INU.

[0147] Also, the transmission unit 132 transmits the input information INU including the auxiliary prompt to the chatbot (step S205).

[0148] In such a state, the acquisition unit 131 determines whether it has acquired the response information ANU generated by the chatbot device 200A for the input information INU (step S206). While the acquisition unit 131 has not acquired the response information ANU (step S206; No), the acquisition unit 131 waits until it can acquire the response information ANU.

[0149] On the other hand, when the transmission unit 132 can acquire the response information ANU (step S206; Yes), it transmits the response information ANU to the user device 10 (step S207). Although not shown in FIG. 8, the output control unit 15b of the user device 10 performs output control for causing the output unit 14 to output the response information ANU.

[0150] <Second Embodiment> Hereinafter, the second embodiment will be described. The information processing according to the first embodiment adjusted the auxiliary prompt according to the evaluation result of evaluating the response accuracy of the chatbot device 200. On the other hand, the information processing according to the second embodiment selects the chatbot to be executed according to the evaluation result of evaluating the response accuracy of the chatbot device 200B, and repeats the selection of the chatbot to be executed until the evaluation result exceeds the passing line.

[0151] As will be described later, in the second embodiment, the selection of the chatbot to be executed includes, for example, either selecting one chatbot device 200 from among the chatbot devices, or selecting a language model to be executed for one chatbot device 200 from among a plurality of candidates.

[0152] [1. Functional Configuration] With reference to FIG. 9, a configuration example of each of the adjustment device 100B and the chatbot device 200B according to the second embodiment will be described. FIG. 9 is a diagram showing a configuration example of the device according to the second embodiment. In FIG. 9, descriptions of points common to the first embodiment are omitted or simplified. For example, since the operation of the user device 10 is the same in both the first embodiment and the second embodiment, the description thereof is omitted. In addition, descriptions of processing units assigned the same reference numerals as those in the first embodiment are also omitted or simplified.

[0153] [Adjustment Device 100B] As shown in FIG. 9, the adjustment device 100B according to the second embodiment includes a communication unit 110, a storage unit 120B, and a control unit 130B.

[0154] (Memory unit 120B) The memory unit 120B is realized by, for example, semiconductor memory elements such as RAM, ROM, and flash memory, or storage devices such as hard disks, SSDs, and optical disks. The memory unit 120B may store, for example, data and programs related to the information processing according to the second embodiment. Also, according to the example of FIG. 2, the memory unit 120B may further include a correct answer information storage unit 121, a prompt information storage unit 122, and a language model information storage unit 123 in addition to these.

[0155] (Language model information storage unit 123) In the information processing according to the second embodiment, an arbitrary language model is selected from among the language model group of selection candidates, and the selected language model is specified for the chatbot device 200B. As a result, the chatbot device 200B generates response information using, for example, the specified language model among the language models registered in the storage unit 220B. Therefore, the language model information storage unit 123 may store information on the language models of the selection candidates, and the stored language models of the selection candidates and the language models registered in the storage unit 220B are in a corresponding relationship as shown in FIG. 9. For example, the information on the language models of the selection candidates may be registered in advance by the administrator T. Note that when physically different accesses to the chatbot device 200B are required for each language model to be used, the language model information storage unit 123 may store the access destination addresses associated with the language models.

[0156] (Control unit 130B) The control unit 130B is realized by, for example, a CPU, an MPU, etc., when various programs (for example, the information processing program according to the second embodiment) stored in the storage device inside the adjustment device 100B are executed using the RAM as a work area. Also, the control unit 130B is realized by, for example, an integrated circuit such as an ASIC or an FPGA.

[0157] As shown in FIG. 9, the control unit 130B includes an acquisition unit 131, a transmission unit 132, a determination information generation unit 133, a determination unit 134B, an execution target adjustment unit 135B, a notification unit 136B, an identification unit 137, and a prompt providing unit 138, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130B is not limited to the configuration shown in FIG. 9, and may be any other configuration as long as it can perform the information processing described later. Also, the connection relationship of each processing unit included in the control unit 130A is not limited to the connection relationship shown in FIG. 2, and may be any other connection relationship.

[0158] (Determination unit 134B) The determination unit 134B determines (evaluates) the response accuracy of the chatbot device 200B based on whether the response information generated by the chatbot device 200B according to the input information satisfies a predetermined condition. Specifically, the determination unit 134B determines whether the first response information generated by the first chatbot device 200B for the input information satisfies a predetermined condition.

[0159] When it is determined that the first response information does not satisfy the predetermined condition and the response accuracy of the first chatbot device 200B is low, an adjustment process is executed to select, as an execution target for generating response information, a second chatbot device 200B having a language model different from that of the first chatbot device 200B. Therefore, the determination unit 134B determines whether the second response information generated by the second chatbot device 200B for the input information satisfies a predetermined condition.

[0160] When it is determined that the second response information does not satisfy the predetermined condition and the response accuracy of the second chatbot device 200B is low, an adjustment process is further executed to select, as an execution target for generating response information, a third chatbot device 200B having a language model different from that of the first chatbot device 200B and the second chatbot device 200B. Therefore, the determination unit 134B further determines whether the third response information generated by the third chatbot device 200B for the input information satisfies a predetermined condition.

[0161] As described above, in the information processing according to the second embodiment, the selection of the execution target is repeated until the condition that the response accuracy of the chatbot device 200B exceeds the passing line is satisfied. Therefore, not limited to the above example, the fourth chatbot device 200B may be selected as the execution target for generating the response information, and further, the fifth chatbot device 200B may be selected as the execution target for generating the response information, and the selection may be repeated in such a manner.

[0162] For example, each time input information of the same content is input, the determination unit 134B calculates the number of characters of each of the first response information generated by the first chatbot device 200B. Then, based on the calculated number of characters, the determination unit 134B calculates a statistical value of the number of characters among a predetermined number of the first response information, and determines whether the statistical value satisfies a predetermined condition.

[0163] As another example, each time input information of the same content is input, the determination unit 134B calculates the similarity between each of the first response information generated by the first chatbot device 200B and the correct answer information prepared in advance for the input information. Then, based on the calculated similarity, the determination unit 134B calculates a statistical value of the similarity among a predetermined number of the first response information, and determines whether the statistical value satisfies a predetermined condition.

[0164] Note that the determination unit 134B may calculate quartiles, mode values, minimum values, etc. as a plurality of types of statistical values, and determine whether more than a predetermined number of the plurality of types of statistical values satisfy a predetermined condition. For example, conditions corresponding to each of the statistical values such as quartiles, mode values, and minimum values may be provided, and the determination unit 134B may evaluate the response accuracy based on whether a predetermined ratio or more of the plurality of types of statistical values satisfy the condition, or whether all of the plurality of types of statistical values satisfy the condition.

[0165] Here, the above example shows a case where the determination unit 134B evaluates the response accuracy of the first chatbot device 200B. However, the determination unit 134B also evaluates the response accuracy of the second chatbot device 200B, the third chatbot device 200B, etc. using the same method.

[0166] (Execution target adjustment unit 135B) When the first response information generated by the first chatbot device 200B for the input information does not satisfy a predetermined condition and it is determined that the response accuracy of the first chatbot device 200B is low, the execution target adjustment unit 135B selects, as an execution target for generating response information, a second chatbot device 200B having a language model different from that of the first chatbot device 200B. Further, when the second response information generated by the second chatbot device 200B for the input information does not satisfy a predetermined condition and it is determined that the response accuracy of the second chatbot device 200B is low, the execution target adjustment unit 135B selects, as an execution target for generating response information, a third chatbot device 200B having a language model different from that of the first chatbot device 200B and the second chatbot device 200B.

[0167] In this way, the adjustment process of selecting an execution target for generating response information from among the chatbot devices 200B is repeated until the condition that the response accuracy of the execution target chatbot device 200B exceeds the pass line is satisfied. Also, the selection of the execution target corresponds to selecting an arbitrary language model from among a group of language models of selection candidates prepared in advance and replacing the language model mounted as a function of the currently executing chatbot device 200B with the selected language model.

[0168] The process of selecting an execution target for generating response information will be specifically described. The execution target adjustment unit 135B selects an arbitrary language model from a group of pre-prepared selection candidate language models, and replaces the language model currently serving as the function of the chatbot with the selected language model. More specifically, when a predetermined condition is not satisfied, the execution target adjustment unit 135B repeatedly performs a replacement process of selecting an arbitrary language model from the group of selection candidate language models and replacing the language model currently serving as the function of the chatbot with the selected language model until the predetermined condition is satisfied.

[0169] In addition, when it is necessary to access physically different chatbot devices 200B for each language model selected as the execution target for generating response information by the execution target adjustment unit 135B, the transmission unit 132 refers to the access destination address for each language model stored in the language model information storage unit 123, transmits the input information IN to the address, and the acquisition unit 131 may acquire the response information generated by the chatbot device 200B for the input information IN.

[0170] (Notification unit 136B) According to the above description, when it is determined that the response information generated by the chatbot device 200B in response to the input information does not meet the predetermined condition and the response accuracy of the chatbot device 200B is low, an adjustment process for automatically improving the response accuracy is executed. Also, as described above, this adjustment process is repeated until the predetermined condition is satisfied, but there is a limit to the number of repetitions of the adjustment process. Therefore, when the predetermined condition is not satisfied even after reaching the limit and the automatic improvement of the response accuracy becomes impossible, the notification unit 136B notifies a predetermined notification destination that the automatic improvement of the response accuracy is impossible. For example, the notification unit 136B may notify the administrator device 30 of the administrator T of the improvement impossible information.

[0171] For example, when all the language models in the selection candidate language model group are selected without the predetermined conditions being satisfied, the notification unit 136B may notify the administrator T that it is impossible to improve the response accuracy.

[0172] [[2. Overall processing procedure of information processing according to the second embodiment]] FIG. 10 is a diagram showing the overall flow of the information processing procedure according to the second embodiment realized by the adjustment device 100B. The information processing according to the second embodiment includes an evaluation process for evaluating the response accuracy of the chatbot device 200B and an adjustment process for selecting an execution target for executing the generation of response information. And these processes are repeated until the response accuracy of the chatbot device 200B satisfies the conditions. Therefore, first, the flow of the first round of the information processing according to the second embodiment will be described, and then the flow of the second round and subsequent rounds of the information processing according to the second embodiment will be described.

[0173] Also, in the example of FIG. 10, a scene where the selection of the execution target is performed in the verification experiment for evaluating the response accuracy of the chatbot device 200B is shown. In such a verification experiment scene, verification input information may be generated by the determination information generation unit 133 at specific timings (for example, every day).

[0174] On the other hand, the evaluation of the response accuracy of the chatbot device 200B and the selection of the execution target may be performed in the actual scene where the chatbot device 200B is utilized by the user U. In such an example, instead of the verification input information generated by the determination information generation unit 133, the input information actually input by the user U in the utilization scene may be used for the evaluation process and the adjustment process.

[0175] Also, there are various types of input information that the chatbot device 200B can handle, such as "route guidance", "cooking recipes", "music content", etc. FIG. 3 shows a scene where the automatic adjustment of the chatbot device 200B in the field of "route guidance" is performed.

[0176] (First round of processing) The determination information generation unit 133 determines whether it is the timing (for example, the timing when it is 16:00 once a day, etc.) to execute an evaluation process for evaluating the response accuracy of the chatbot device 200B (step S301). When it is not the timing to execute the evaluation process (step S301; No), the determination information generation unit 133 waits until it is the timing to execute the evaluation process.

[0177] On the other hand, when it is the timing to execute the evaluation process (step S301; Yes), the determination information generation unit 133 generates input information for verification based on a fixed scenario (step S302). In the example of FIG. 10 in which the automatic adjustment of the chatbot device 200B in the field of "route guidance" is performed, it is assumed that the determination information generation unit 133 generates input information IN such as "I want to go to the Tokyo Tower" using the fixed scenario "I want to go to XX". Note that the determination information generation unit 133 may generate input information with the same content each time.

[0178] Next, the execution target adjustment unit 135B determines whether the adjustment process has not been executed at the current time (step S303). In the first round of processing, the execution target adjustment unit 135B determines that the adjustment process has not been executed at the current time (step S303; not executed), and initially designates a language model (step S304). For example, the execution target adjustment unit 135B may select one arbitrary language model from the group of candidate language models stored in the language model information storage unit 123 and initially designate the selected language model. In the example of FIG. 10, it is assumed that the execution target adjustment unit 135B selects the language model LLM1 and initially designates the language model LLM1.

[0179] The transmission unit 132 transmits to the chatbot device 200B information for setting the language model LLM1 specified by the adjustment unit 135B to be executed as the language model to be used in subsequent conversations. Alternatively, each time the transmission unit 132 has an opportunity to transmit the input information IN to the chatbot device 200B, it may transmit information for setting the language model LLM1 as the language model for conversation prior to the input information IN. Then, the transmission unit 132 transmits the input information IN to the chatbot CB1 (an example of the first chatbot device 200B) equipped with the language model LLM1 (step S305). The chatbot CB1 applies the language model LLM1 to the input information IN and generates response information AN1 based on the output result by the language model LLM1.

[0180] Note that an auxiliary prompt may be added to the input information IN transmitted to the chatbot CB1. For example, the prompt addition unit 138 may select one arbitrary auxiliary prompt from the group of auxiliary prompts for change candidates stored in the prompt information storage unit 122 and add it to the input information IN generated in step S302.

[0181] Also, in the information processing according to the second embodiment, the auxiliary prompt given at each turn may be fixed. On the other hand, in the information processing according to the second embodiment, an optimal combination (combination of auxiliary prompt and language model) evaluated to have high response accuracy of the chatbot device 200 may be searched by giving different auxiliary prompts at each turn. That is, in the information processing according to the second embodiment, the information processing according to the first embodiment may also be carried out in parallel.

[0182] Returning to the description of FIG. 10, the acquisition unit 131 acquires the response information AN1 generated by the chatbot CB1 for the input information IN (step S306).

[0183] The determination unit 134B determines whether a predetermined number of response information AN1 generated by the chatbot CB1 for the input information IN has been accumulated (for example, 100 are accumulated) (step S307). If the determination unit 134B determines that the response information AN1 has not been accumulated by a predetermined number (step S307; No), the process returns to step S302 and is repeated until the response information AN1 is accumulated by a predetermined number.

[0184] On the other hand, when the determination unit 134B determines that the response information AN1 has been accumulated by a predetermined number (step S307; Yes), it executes an evaluation value calculation process for the response accuracy using the predetermined number of response information AN1 (step S308). The evaluation process performed in step S308 is the same as the pattern of the first embodiment described in FIGS. 4 and 5, so the description is omitted. Here, although each of the response information AN1 accumulated by a predetermined number is a response to the input information IN of the same content, due to the response variation of the chatbot CB1, there may be responses of different contents. Therefore, the determination unit 134B evaluates the response accuracy of the chatbot CB1 based on whether the evaluation value calculated in step S308 satisfies a predetermined condition (step S309).

[0185] When the evaluation value of the response accuracy satisfies a predetermined condition (step S309; Yes), the execution target adjustment unit 135B registers the currently executed language model (for example, the language model LLM1) as a generation algorithm capable of realizing a chatbot device 200B with high response accuracy (step S310). That is, the execution target adjustment unit 135B registers a chatbot having a language model verified to have high response accuracy as an execution target for generating response information. Then, the process ends.

[0186] On the other hand, when the execution target adjustment unit 135B determines that the evaluation value of the response accuracy does not satisfy a predetermined condition (step S309; No), it executes an adjustment process for selecting the language model to be executed (step S313). The detailed procedure of the adjustment process performed in step S313 will be described with reference to FIG. 11.

[0187] After the adjustment process is performed by the execution target adjustment unit 135B, the process returns to step S302 and proceeds to the second pass of the process.

[0188] (After the second pass of the process) After the second pass of the process, the determination information generation unit 133 generates input information for verification based on a fixed scenario (step S302). The determination information generation unit 133 may generate the same input information "I want to go to the Tokyo Tower." as in the first pass of the process for the second pass and subsequent passes.

[0189] Next, the execution target adjustment unit 135B determines whether the adjustment process has not been executed at the current time (step S303). In the case of the second pass and subsequent passes, the execution target adjustment unit 135B determines that the adjustment process has not been executed at the current time, that is, it has been executed (step S303; executed), and designates the language model selected as the execution target by the adjustment process in step S313 (step S314). Here, it is assumed that the language model LLM2 is selected in the adjustment process of step S313.

[0190] The transmission unit 132 transmits the input information IN to the chatbot CB2 (an example of the second chatbot device 200B) equipped with the language model LLM2 (step S315). As described above, the input information IN transmitted to the chatbot CB2 may be provided with an auxiliary prompt having the same content as in the first pass of the process, or an auxiliary prompt having different content may be provided.

[0191] The chatbot CB2 applies the language model LLM2 to the input information IN and generates response information AN2 based on the output result by the language model LLM2.

[0192] The acquisition unit 131 acquires the response information AN2 generated by the chatbot CB2 for the input information IN (step S316).

[0193] The determination unit 134B determines whether a predetermined number of response information AN2 generated by the chatbot CB2 for the input information IN has been accumulated (for example, 100 are accumulated) (step S307). If the determination unit 134B determines that the response information AN2 has not been accumulated by a predetermined number (step S307; No), the process returns to step S302 and is repeated until the response information AN2 is accumulated by a predetermined number.

[0194] On the other hand, when the determination unit 134A determines that the response information AN2 has been accumulated by a predetermined number (step S307; Yes), it executes a process for calculating an evaluation value of the response accuracy using the predetermined number of response information AN2 (step S308). Here, although each of the response information AN2 accumulated by a predetermined number is a response to the input information IN of the same content, due to the response variation of the chatbot CB2, there may be responses of different contents. Therefore, the determination unit 134B evaluates the response accuracy of the chatbot CB2 based on whether the evaluation value calculated in step S308 satisfies a predetermined condition (step S309).

[0195] When the evaluation value of the response accuracy satisfies a predetermined condition (step S309; Yes), the execution target adjustment unit 135B registers the currently executed language model (for example, the language model LLM2) as a generation algorithm capable of realizing a chatbot device 200B with high response accuracy (step S310). Here, in the past process, the currently executed language model (for example, the language model LLM2) is newly registered by replacing it with a language model (for example, the language model LLM1) registered as a generation algorithm capable of realizing a chatbot device 200B with high response accuracy. That is, the execution target adjustment unit 135B registers a chatbot having a language model verified to have high response accuracy as an execution target for executing the generation of response information. Then, the process ends.

[0196] On the other hand, even at the current point after the second round of processing, there may be no language model that can obtain an evaluation value satisfying the predetermined conditions. When it is determined that the evaluation value of the response accuracy does not satisfy the predetermined conditions (step S309; No), the execution target adjustment unit 135B executes again the adjustment process of selecting a language model (step S313).

[0197] After the adjustment process is performed by the execution target adjustment unit 135B, the process returns to step S302, and the process proceeds to the next round.

[0198] [3. Specific Procedures of the Adjustment Process] Subsequently, the specific procedures of the adjustment process (the adjustment process of selecting the language model to be executed) performed in step S313 of FIG. 10 will be described.

[0199] FIG. 11 is a diagram showing the specific procedures of the adjustment process according to the second embodiment. The execution target adjustment unit 135B determines whether there is an unselected language model among the group of language models prepared in advance (step S3131).

[0200] When there is an unselected language model among the group of language models prepared in advance (step S3131; Yes), the execution target adjustment unit 135B selects one of the unselected language models (step S3132). For example, assume that the language model LLM1 is currently the execution target, and there are language models MC31 and MC32 as shown in FIG. 11 as unselected language models. In such an example, the execution target adjustment unit 135B can select either one of the language models MC31 and MC32.

[0201] Then, the execution target adjustment unit 135B exchanges the currently executed language model with the language model selected in step S3132 (step S3133).

[0202] Subsequently, the execution target adjustment unit 135B holds the replaced language model as the language model selected as the execution target by the adjustment process (step S3134). Then, the process proceeds to step S301. For example, assume that the language model LLM1 is replaced with the language model LLM2 listed as a candidate. In such an example, the language model LLM2 is held as the language model selected as the execution target by the adjustment process and will be specified in step S314 of FIG. 10.

[0203] On the other hand, if there is no unselected language model among the group of pre-prepared language models (step S3131; No), that is, if the pre-prepared language models are all used up, the execution target adjustment unit 135B recognizes that no response information can be generated to obtain an evaluation value that satisfies the conditions no matter which language model is installed in the chatbot device 200B (the response accuracy of the chatbot device 200B has not been improved no matter which language model is used) (step S3135).

[0204] Then, the notification unit 136B notifies the administrator T that it is impossible to improve the response accuracy of the chatbot device 200B (step 3136).

[0205] <Other Embodiments> The information processing according to each of the above-described embodiments is not limited to application in a PULL-type system such as a chatbot, but can be extended to a PUSH-type system such as automatic announcement to users. For example, the case where the information processing according to each embodiment is applied to an automatic announcement system considering the surrounding environment will be described as an example. In such a case, the adjustment device 100 acquires the surrounding traffic situation as a parameter in conjunction with the current location, and prompts the acquired parameter and inputs it to the chatbot device 200. The chatbot device 200 may generate response information for various driving supports according to the traffic situation.

[0206] From the above, by extending the information processing according to each embodiment to a PUSH-type automatic announcement system, it becomes possible to realize a system that does not require the user to input a prompt and spontaneously provides user support including attention calls according to the situation.

[0207] Note that even when the information processing according to each embodiment is extended to a PUSH-type automatic announcement system, for example, there may be cases where the response information is encouraging or the response information includes words unrelated to the gist. Thus, even when it can be evaluated that the response accuracy of the chatbot device 200 has decreased in the PUSH-type service, it is possible to realize automatic improvement of the response accuracy.

[0208] <Hardware Configuration> The adjustment device 100 described above may be realized by, for example, a computer 1000 configured as shown in FIG. 12. FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the adjustment device 100 according to the embodiment. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0209] The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program dependent on the hardware of the computer 1000, and the like.

[0210] The HDD 1400 stores a program executed by the CPU 1100 and data used by such a program. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and sends data generated by the CPU 1100 to other devices via a predetermined communication network.

[0211] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

[0212] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory or the like.

[0213] For example, when the computer 1000 functions as the adjustment device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing the program loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800. As another example, these programs may be acquired from another device via a predetermined communication network.

[0214] <Others> In addition, among the processes described in each of the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, regarding the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0215] In addition, each component of each device shown in the drawings is a functional concept, and it is not necessarily physically configured as shown in the drawings. That is, the specific form of the distribution and integration of each device is not limited to that shown in the drawings, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage situations, etc.

[0216] In addition, the above embodiments can be appropriately combined within a range that does not conflict with the processing content.

[0217] As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are merely examples, and the present invention can be implemented in other forms with various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the column of the present invention.

Explanation of Reference Numerals

[0218] 1 System 10 User Device 30 Administrator Device 100A Adjustment Device 134A Determination Unit 135A Prompt Adjustment Unit 136A Notification Unit 100B Adjustment Device 134B Determination Unit 135B Execution Target Adjustment Unit 136B Notification Unit 200 Chatbot Device

Claims

1. A determination unit that determines whether or not first response information generated by a first chatbot for input information satisfies a predetermined condition; An adjustment unit that, when the determination unit determines that the first response information does not satisfy the predetermined condition, selects, as an execution target, a second chatbot having a language model different from that of the first chatbot to execute generation of response information; An information processing apparatus comprising the above.

2. The determination unit determines whether or not second response information generated by the second chatbot for the input information satisfies the predetermined condition, The adjustment unit, when the determination unit determines that the second response information does not satisfy the predetermined condition, selects, as the execution target, a third chatbot having a language model different from that of the first chatbot and the second chatbot The information processing apparatus according to claim 1.

3. The determination unit calculates a statistical value of the number of characters among a predetermined number of the first response information based on the number of characters of each of the first response information generated by the first chatbot each time the input information of the same content is input, and determines whether or not the statistical value satisfies the predetermined condition The information processing apparatus according to claim 1.

4. The determination unit calculates a statistical value of the similarity among a predetermined number of the first response information based on the similarity between each of the first response information generated by the first chatbot each time the input information of the same content is input and correct answer information prepared in advance for the input information, and determines whether or not the statistical value satisfies the predetermined condition The information processing apparatus according to claim 1.

5. The determination unit calculates a plurality of types of the statistical values, and determines whether or not more than a predetermined number of the statistical values among the plurality of types of the statistical values satisfy the predetermined condition The information processing apparatus according to claim 3 or 4.

6. Further comprising a determination information generation unit that generates input information of a predetermined content, which is input information used for determining whether or not the input information satisfies the predetermined condition, as the input information, The determination information generation unit generates the input information used for the determination a plurality of times at preset timings. The information processing apparatus according to claim 3 or 4.

7. The adjustment unit, ​ When it is evaluated that the first response information does not satisfy the predetermined condition and the response accuracy of the first chatbot is low, the second chatbot having a language model different from that of the first chatbot is selected as an execution target for executing the generation of response information. When it is evaluated that the second response information does not satisfy the predetermined condition and the response accuracy of the second chatbot is low, a third chatbot having a language model different from that of the first chatbot and the second chatbot is selected. The information processing apparatus according to claim 2.

8. As a process of selecting an execution target for executing the generation of response information, the adjustment unit selects an arbitrary language model from a group of language models of selection candidates prepared in advance, and replaces the language model mounted as a function of the currently executing chatbot with the selected language model. The information processing apparatus according to claim 7.

9. When the predetermined condition is not satisfied, the adjustment unit selects an arbitrary language model from the group of language models of selection candidates until the predetermined condition is satisfied, and repeatedly performs a replacement process of replacing the language model mounted as a function of the currently executing chatbot with the selected language model. The information processing apparatus according to claim 8.

10. When all the language models have been selected from the group of language models of selection candidates while the predetermined condition is not satisfied, the adjustment unit notifies the administrator that it is impossible to improve the response accuracy. The information processing apparatus according to claim 9.

11. An information processing method executed by an information processing apparatus, a determination step of determining whether first response information generated by a first chatbot for input information satisfies a predetermined condition; an adjustment step of selecting, as an execution target for executing the generation of response information, a second chatbot having a language model different from that of the first chatbot when it is determined by the determination step that the first response information does not satisfy the predetermined condition; An information processing method including the above.

12. An information processing program executed by an information processing apparatus, a determination procedure of determining whether first response information generated by a first chatbot for input information satisfies a predetermined condition; When it is determined by the determination procedure that the first response information does not satisfy the predetermined condition, an adjustment procedure for selecting, as an execution target for executing generation of response information, a second chatbot having a language model different from that of the first chatbot; An information processing program for causing the information processing apparatus to execute the above.

Images