Information processing device, information processing method, and information processing program

Abstract

An information processing device according to the present invention comprises a determination unit, an adjustment unit, and a notification unit. The determination unit acquires response information generated by a chatbot in accordance with input information, and determines response accuracy of the chatbot on the basis of whether or not the acquired response information satisfies a prescribed condition. When the response information does not satisfy the prescribed condition and the response accuracy of the chatbot is determined to be low, the adjustment unit executes prescribed processing for automatically improving the response accuracy. When the response information acquired by the determination unit does not satisfy the prescribed condition after the prescribed processing has been executed, the notification unit notifies a prescribed notification destination that the automatic improvement of the response accuracy is impossible.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] In recent years, various voice assistance functions using chatbots equipped with large-scale language models have been proposed.

[0003] Meanwhile, Patent Document 1 discloses a technique for requesting feedback from a user regarding one or more content parameters of a suggestion or other content provided by a chatbot.

[0004] Special table 2019-537802 publication

[0005] However, the above-mentioned conventional technologies focus on the content of suggestions, such as improving future suggestions and other content provided to users, and do not take into consideration improving the accuracy of responses provided by chatbots.

[0006] Therefore, the above-mentioned conventional techniques do not necessarily make users understand that human intervention is necessary to improve the response accuracy of chatbots.

[0007] The present invention has been made in consideration of the above, and proposes an information processing device, an information processing method, and an information processing program that can make users understand that human intervention is required to improve the response accuracy of chatbots.

[0008] The information processing device described in claim 1 comprises a judgment unit that acquires response information generated by a chatbot in response to input information and judges the response accuracy of the chatbot based on whether the acquired response information satisfies predetermined conditions; an adjustment unit that executes predetermined processing to automatically improve the response accuracy if the response information does not satisfy the predetermined conditions and the response accuracy of the chatbot is judged to be low; and a notification unit that notifies a predetermined notification destination that automatic improvement of the response accuracy is not possible if the response information acquired by the judgment unit does not satisfy the predetermined conditions after the adjustment unit executes the predetermined processing.

[0009] The information processing method described in claim 9 is an information processing method executed by an information processing device, and includes: a determination step of acquiring response information generated by a chatbot in response to input information, and determining the response accuracy of the chatbot based on whether the acquired response information satisfies predetermined conditions; an adjustment step of executing predetermined processing to automatically improve the response accuracy if the response information does not satisfy the predetermined conditions and the response accuracy of the chatbot is determined to be low; and a notification step of notifying a predetermined notification destination that automatic improvement of the response accuracy is not possible if the response information acquired by the determination step does not satisfy the predetermined conditions after the adjustment step has executed the predetermined processing.

[0010] The information processing program described in claim 10 is an information processing program executed by an information processing device, and causes the information processing device to execute the following steps: a determination step for acquiring response information generated by a chatbot in response to input information, and determining the response accuracy of the chatbot based on whether the acquired response information satisfies predetermined conditions; an adjustment step for executing predetermined processing to automatically improve the response accuracy if the response information does not satisfy the predetermined conditions and the response accuracy of the chatbot is determined to be low; and a notification step for notifying a predetermined notification destination that automatic improvement of the response accuracy is not possible if the response information acquired by the determination step does not satisfy the predetermined conditions after the adjustment step has executed the predetermined processing.

[0011] FIG. 1 is a diagram illustrating an example of a system according to an embodiment. FIG. 2 is a diagram illustrating an example of a device configuration according to the first embodiment. FIG. 3 is a diagram illustrating an overall flow of an information processing procedure according to the first embodiment, which is realized by an adjustment device. FIG. 4 is a diagram illustrating a specific procedure (1) of an evaluation process according to an embodiment. FIG. 5 is a diagram illustrating a specific procedure (2) of an evaluation process according to an embodiment. FIG. 6 is a diagram illustrating a specific procedure (1) of an adjustment process according to the first embodiment. FIG. 7 is a diagram illustrating a specific procedure (2) of an adjustment process according to an embodiment. FIG. 8 is a diagram illustrating a dialogue control procedure executed by an adjustment device according to the first embodiment. FIG. 9 is a diagram illustrating an example of a device configuration according to a second embodiment. FIG. 10 is a diagram illustrating an overall flow of an information processing procedure according to the second embodiment, which is realized by an adjustment device. FIG. 11 is a diagram illustrating a specific procedure of an adjustment process according to the second embodiment. FIG. 12 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an adjustment device according to an embodiment.

[0012] [Embodiments] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects.

[0014] 1. Introduction For example, a service can be realized in which a terminal device receives an answer corresponding to an input from a chatbot that works with a large-scale language model via an API (Application Programming Interface) and displays the answer on the terminal device.

[0015] However, even if the same instruction is input to a chatbot, a response with a different meaning may be output. For example, if a user inputs "I want to go to Tokyo Tower," the chatbot may respond with a content that is in line with the user's intention to receive directions, such as "The nearest station is Station A. It is accessible on foot from Station A." On the other hand, if a user similarly inputs "I want to go to Tokyo Tower" at a different time, the chatbot may respond with a content that is different in meaning from directions, such as "Tokyo Tower is a famous tourist spot in Japan."

[0016] In this way, the responses generated by the chatbot may be inconsistent and uneven, which is called response fluctuation. Response fluctuation can also be considered as unnatural response generation.

[0017] In order to suppress response fluctuation, prompts with specific content may be set. However, for example, when a large-scale language model undergoes new learning and evolves or is upgraded, new response fluctuations may occur even if prompts with the same content are set. In this case, the service provider is forced to revise the content of the prompts whenever response fluctuations, i.e., responses with content that differs from the intended meaning, increase.

[0018] The present invention proposes a mechanism that can improve the response accuracy of a chatbot by automatically adjusting the degradation in the consistency (appropriateness) of the response content of a large-scale language model when that degradation occurs.

[0019] However, if automatic adjustment has limitations in improving response accuracy, intervention by the service provider is required. However, as mentioned above, it is cumbersome for the service provider to have to perform frequent correction work. Therefore, the present invention proposes a mechanism that notifies the service provider when automatic adjustment is no longer able to improve response accuracy and manual improvement is required.

[0020] [2. System Configuration] First, the configuration of a system according to an embodiment will be described using Fig. 1. Although the following description will be divided into multiple embodiments, the system shown in Fig. 1 is common to all embodiments. Fig. 1 is a diagram showing an example of a system according to an embodiment. Fig. 1 shows system 1 as an example of a system according to an embodiment. Information processing according to the proposed technology of the present invention is realized in system 1.

[0021] 1, system 1 includes user devices 10, administrator devices 30, adjustment devices 100, and chatbot devices 200. The user devices 10, administrator devices 30, adjustment devices 100, and chatbot devices 200 are communicably connected via a network N, either wired or wirelessly. The number of user devices 10, administrator devices 30, adjustment devices 100, and chatbot devices 200 in system 1 is not limited.

[0022] The user device 10 may be an information processing terminal used by a user U who wishes to receive information through a dialogue with a voice assistant. For example, the user device 10 is a smartphone, a wearable device, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like.

[0023] As another example, the user device 10 may be implemented as a navigation device built into or mounted on a vehicle, i.e., an in-vehicle device. The user device 10 as an in-vehicle device may have not only a navigation function but also a recording function (a drive recorder function).

[0024] The administrator device 30 may be an information processing terminal used by an administrator T who manages the entire system 1 and the maintenance of the chatbot device 200. For example, the administrator device 30 is a smartphone, a wearable device, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, or the like.

[0025] The adjustment device 100 is an example of an information processing device. The adjustment device 100 is an information processing device that performs, as information processing according to the embodiment, an adjustment process for restoring consistency to response content when response fluctuation occurs in a chatbot, and a notification process for notifying that manual improvement is necessary when automatic improvement of response accuracy through the adjustment process is not possible. The functions of the adjustment device 100 may be realized by an information processing program according to the embodiment.

[0026] The chatbot device 200 serves as a voice assistant and realizes a dialogue with the user U. The chatbot device 200 has a function of generating response information to input information by incorporating a language model. For example, the chatbot device 200 repeatedly learns the language model and updates the language model so that it can realize a highly accurate dialogue with the user U.

[0027] The chatbot device 200 may be equipped with so-called generative AI (artificial intelligence that can create various content and ideas such as conversations, stories, images, videos, and music). In the following embodiments, the term "chatbot" essentially refers to the "chatbot device 200 (200A, 200B)."

[0028] Furthermore, while the user device 10 and the administrator device 30 are edge computers, the adjustment device 100 and the chatbot device 200 can be implemented as cloud computers.

[0029] In the following, the embodiments will be described separately as a first embodiment and a second embodiment. However, when there is no need to distinguish between the first embodiment and the second embodiment, the information processing according to each embodiment will simply be referred to as "information processing according to the embodiment."

[0030] Furthermore, the adjustment device 100 according to the first embodiment will be referred to as "adjustment device 100A," and the adjustment device 100 according to the second embodiment will be referred to as "adjustment device 100B." When there is no need to distinguish between the "adjustment device 100A" and the "adjustment device 100B," they will simply be referred to as "adjustment device 100."

[0031] Furthermore, the chatbot device 200 according to the first embodiment will be referred to as the "chatbot device 200A," and the chatbot device 200 according to the second embodiment will be referred to as the "chatbot device 200B." When there is no need to distinguish between the "chatbot device 200A" and the "chatbot device 200B," they will simply be referred to as the "chatbot device 200."

[0032] <First embodiment> [1. Functional configuration] An example configuration of each of the user device 10, adjustment device 100A, and chatbot device 200A according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example device configuration according to the first embodiment. The administrator device 30 is omitted from Fig. 2.

[0033] [User Device 10] As shown in FIG. 2, the user device 10 according to the first embodiment includes a communication unit 11, a storage unit 12, an input unit 13, an output unit 14, and a control unit 15.

[0034] (Communication Unit 11) The communication unit 11 is realized by, for example, a network interface card (NIC), etc. The communication unit 11 is connected to the network N via a wired or wireless connection, and transmits and receives information to and from, for example, the adjustment device 100 and the chatbot device 200.

[0035] (Storage unit 12) The storage unit 12 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as a hard disk, an SSD (Solid State Drive), or an optical disk. The storage unit 12 may store, for example, various data related to the information processing according to the embodiment and input information.

[0036] (Input Unit 13) The input unit 13 is an input device that accepts various inputs from the outside. For example, the input unit 13 is an operation device such as a keyboard, a mouse, or operation keys that the user U uses to perform various operations. If a touch panel is employed in the user device 10, the touch panel is also included in the input unit 13. In this case, the user U performs various operations by touching the touch panel. The input unit 13 also includes a microphone that accepts voice input by speaking.

[0037] The user U may input various types of input information via the input unit 13, such as, for example, "I want to go to XX," "I want to buy XX," "I want to eat XX," or "I want to hear XX." The input information may be text or audio. The input information may be input to the chatbot device 200 via the adjustment device 100.

[0038] (Output unit 14) The output unit 14 is a device that outputs various types of information to the outside, such as sound, light, vibration, and image. The output unit 14 outputs various types of information to the user U under the control of the control unit 15. The output unit 14 may be a display device that displays various types of information. The display device is, for example, a liquid crystal display or an organic electroluminescence display (OLED). The output unit 14 may be a touch panel display device. In this case, the input unit 13 and the output unit 14 may be considered to be an integrated configuration. The output unit 14 may also be a speaker.

[0039] (Control Unit 15) The control unit 15 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in a storage device inside the user device 10 using RAM as a work area. The control unit 15 is also realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0040] As shown in Fig. 2, the control unit 15 has a transmission / reception unit 15a and an output control unit 15b, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in Fig. 2, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units of the control unit 15 are not limited to the connection relationships shown in Fig. 2, and may be other connection relationships.

[0041] (Transmitter / receiver 15a) The transmitter / receiver 15a receives input information input via the input unit 13. For example, the transmitter / receiver 15a receives voice input via a microphone or touch input via a touch panel. The transmitter / receiver 15a then transmits the received input information. For example, the transmitter / receiver 15a may transmit the received input information to the adjustment device 100, or may transmit the received input information directly to the chatbot device 200.

[0042] The transmitter / receiver 15a also receives response information generated by the chatbot device 200 in response to input information.

[0043] (Output control unit 15b) The output control unit 15b performs output control to cause the output unit 14 to output the response information generated by the chatbot device 200.

[0044] [Adjustment Device 100A] As shown in FIG. 2, the adjustment device 100A according to the first embodiment includes a communication unit 110, a storage unit 120A, and a control unit 130A.

[0045] (Communication Unit 110) The communication unit 110 is realized by, for example, a NIC etc. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information between, for example, the user device 10, the administrator device 30, and the chatbot device 200.

[0046] (Storage Unit 120A) The storage unit 120A is realized by, for example, a semiconductor memory element such as RAM, ROM, or flash memory, or a storage device such as a hard disk, SSD, or optical disk. The storage unit 120A may store, for example, data and programs related to the information processing according to the first embodiment. Furthermore, according to the example of FIG. 2, the storage unit 120A may include a correct answer information storage unit 121 and a prompt information storage unit 122.

[0047] (Correct Answer Information Storage Unit 121) The correct answer information storage unit 121 stores correct answer response information (correct answer information) prepared for input information. The correct answer response information may be prepared in advance by the administrator T, for example.

[0048] (Prompt Information Storage Unit 122) The prompt information storage unit 122 stores various prompts according to the first embodiment. Here, a prompt will be explained. A prompt indicates an instruction sentence used to ask a question or give an instruction to the chatbot device 200, and corresponds to at least a part of the input information input to the chatbot device 200.

[0049] For example, input information may be composed of a wording that is the gist of an instruction and a wording that supports the instruction. In the following embodiment, the wording that supports the instruction is defined as an "assistance prompt." A specific example of this point will be given. For example, assume that user U is a driver and the user U utters and inputs "I want to go to Tokyo Tower" into the input unit 13. In this case, the adjustment device 100 performs an intention interpretation based on the text "I want to go to Tokyo Tower," and identifies the type of input information (in this example, directions) according to the intention interpretation result. Then, the adjustment device 100 provides an assistance prompt according to the type of input information.

[0050] As an example, "Please respond as the driver's assistant" may be assigned as an auxiliary prompt. Here, "driver's assistant" refers to the role that the adjustment device 100, which has interpreted the intention of the user U's speech input "I want to go to Tokyo Tower," as route guidance, determines that the chatbot device 200 should provide in the route guidance situation. In this example, "The driver wants to go to Tokyo Tower." and "Please respond as the driver's assistant" are input information, i.e., prompts, and the part included therein, "Please respond as the driver's assistant," is the auxiliary prompt. On the other hand, the part other than the auxiliary prompt, specifically "The driver wants to go to Tokyo Tower," may be the prompt, or the entire "The driver wants to go to Tokyo Tower." and "Please respond as the driver's assistant" may be the prompt.

[0051] Returning to the description of Fig. 1, the prompt information storage unit 122 may store a group of candidate assistance prompts to be used by the adjustment device 100. As will be described later, the adjustment device 100 may dynamically generate input information to evaluate the response accuracy of the chatbot device 200. Therefore, the prompt information storage unit 122 may also store scenario information for dynamically generating input information (prompts).

[0052] Here, the information processing according to the first embodiment includes an adjustment process in which the above-mentioned assistance prompts are adjusted according to the evaluation results of the response accuracy of the chatbot device 200, and the adjustment of the assistance prompts is repeated until the evaluation results exceed the pass line.

[0053] (Control unit 130A) The control unit 130A is realized by a CPU, an MPU, etc., executing various programs (e.g., the information processing program according to the first embodiment) stored in a storage device inside the adjustment device 100A using RAM as a work area. The control unit 130A is also realized by an integrated circuit such as an ASIC or FPGA.

[0054] As shown in FIG. 2 , the control unit 130A has an acquisition unit 131, a transmission unit 132, a determination information generation unit 133, a determination unit 134A, a prompt adjustment unit 135A, a notification unit 136A, an identification unit 137, and a prompt assignment unit 138, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130A is not limited to the configuration shown in FIG. 2 , and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units of the control unit 130A are not limited to the connection relationships shown in FIG. 2 , and may be other connection relationships.

[0055] (Acquisition unit 131) The acquisition unit 131 acquires or receives various types of information in the information processing according to the embodiment. For example, the acquisition unit 131 acquires input information to be input to the chatbot device 200. For example, the acquisition unit 131 acquires input information received by the input unit 13 as input information to be input to the chatbot device 200. The acquisition unit 131 may also acquire input information generated by the determination information generation unit 133.

[0056] The acquisition unit 131 acquires correct response information (correct answer information) prepared for input information.

[0057] The acquisition unit 131 acquires response information generated by the chatbot device 200 in response to input information. The acquisition unit 131 also acquires an executed assistance prompt that has undergone an adjustment process for adjusting the assistance prompt. Furthermore, the acquisition unit 131 may also acquire information about a language model.

[0058] (Transmitting unit 132) The transmitting unit 132 transmits various types of information in the information processing according to the embodiment. For example, the transmitting unit 132 transmits input information including a help prompt to the chatbot device 200. The transmitting unit 132 also transmits response information generated by the chatbot device 200 in response to the input information to the user device 10.

[0059] (Determination Information Generator 133) The determination information generator 133 generates input information. For example, in a process of determining whether response information generated by the chatbot device 200 satisfies a predetermined condition (i.e., an evaluation process of evaluating the response accuracy of the chatbot device 200), input information for verifying the response accuracy is periodically input to the chatbot device 200. Therefore, the determination information generator 133 may generate this input information for verification at predetermined intervals, for example.

[0060] For example, the determination information generation unit 133 can generate input information according to fixed scenarios such as "I want to go to XX," "I want to buy XX," "I want to eat XX," "I want to hear about XX," etc. For example, when it is desired to evaluate the response accuracy of the chatbot device 200 in response to input information of the type of directions, the determination information generation unit 133 generates input information based on the fixed scenario "I want to go to XX."

[0061] (Determination unit 134A) The determination unit 134A determines (evaluates) the response accuracy of the chatbot device 200A based on whether or not the response information generated by the chatbot device 200A in response to input information satisfies a predetermined condition. Specifically, the determination unit 134A determines whether or not the first response information generated by the chatbot device 200A in response to first input information satisfies a predetermined condition.

[0062] If the first response information does not satisfy the predetermined condition and the chatbot device 200A is determined to have low response accuracy, an adjustment process is performed to adjust the assistance prompt included in the first input information. Therefore, the determination unit 134A may further determine whether the second response information generated by the chatbot device 200A in response to the second input information including the assistance prompt for which the adjustment process has been performed satisfies the predetermined condition.

[0063] For example, the determination unit 134A calculates the number of characters in each piece of first response information generated by the chatbot device 200A each time the chatbot device 200A receives first input information with the same content. Then, the determination unit 134A calculates a statistical value of the number of characters among a predetermined number of pieces of first response information based on the calculated number of characters, and determines whether the statistical value satisfies a predetermined condition.

[0064] As another example, each time the chatbot 200A receives the same first input information, the determination unit 134A calculates the similarity between each piece of first response information generated by the chatbot 200A and the correct answer information prepared in advance for the first input information. Then, the determination unit 134A calculates a statistical value of the similarity between a predetermined number of pieces of first response information based on the calculated similarity, and determines whether the statistical value satisfies a predetermined condition.

[0065] The determination unit 134A may calculate quartiles, modes, minimum values, etc. as the multiple types of statistical values, and determine whether or not a predetermined number of the multiple types of statistical values ​​satisfy a predetermined condition. For example, a corresponding condition may be set for each statistical value, such as the quartiles, modes, and minimum values, and the determination unit 134A may evaluate the response accuracy based on whether or not a predetermined percentage or more of the multiple types of statistical values ​​satisfy the condition, or whether all of the multiple types of statistical values ​​satisfy the condition.

[0066] (Prompt Adjustment Unit 135A) If the first response information does not satisfy a predetermined condition and it is determined that the response accuracy of the chatbot device 200A is low, the prompt adjustment unit 135A adjusts the assistance prompt included in the first input information. Furthermore, if the second response information including the assistance prompt for which the adjustment process has been performed does not satisfy a predetermined condition and it is determined that the response accuracy of the chatbot device 200A is still low, the prompt adjustment unit 135A again adjusts the assistance prompt included in the second input information.

[0067] In this way, the adjustment process for adjusting the assistance prompt is repeated until the condition that the response accuracy of the chatbot device 200A exceeds the pass line is met. Furthermore, adjusting the assistance prompt corresponds to changing the assistance prompt. Taking these factors into consideration, the adjustment process for adjusting the assistance prompt corresponds to changing the assistance prompt currently being adjusted to another assistance prompt.

[0068] Furthermore, the process of changing the currently adjusted assist prompt to another assist prompt is classified into a process of changing the entire assist prompt and a process of changing only some of the words included in the assist prompt.

[0069] The process of changing the entire assistance prompt will be described in detail below. The prompt adjustment unit 135A selects an arbitrary assistance prompt from a group of assistance prompt candidates prepared in advance according to the type of input information to the chatbot device 200A, and executes a process of replacing the currently-adjusted assistance prompt with the selected assistance prompt. More specifically, if a predetermined condition is not satisfied, the prompt adjustment unit 135A repeats the process of selecting an arbitrary assistance prompt from the group of assistance prompt candidates and replacing the currently-adjusted assistance prompt with the selected assistance prompt until the predetermined condition is satisfied.

[0070] Next, a process for changing only some of the words included in the assistance prompt will be described in detail. The prompt adjustment unit 135A changes a predetermined word included in the assistance prompt currently being adjusted to another word, and replaces the assistance prompt currently being adjusted with a changed assistance prompt in which the predetermined word has been replaced with the other word. More specifically, if a predetermined condition is not satisfied, the prompt adjustment unit 135A repeats the replacement process of changing a predetermined word included in the assistance prompt currently being adjusted to another word and replacing the assistance prompt currently being adjusted with the changed assistance prompt in which the predetermined word has been replaced with the other word, until the predetermined condition is satisfied.

[0071] For example, the prompt adjustment unit 135A changes a word that specifies a role according to the type of input information to the chatbot device 200 as a specified word contained in the auxiliary prompt currently being adjusted to a word that specifies another role that is similar to the role in question.

[0072] (Notification Unit 136A) According to the description above, if the response information generated by the chatbot device 200A in response to input information does not satisfy a predetermined condition and the response accuracy of the chatbot device 200A is determined to be low, an adjustment process is executed to automatically improve the response accuracy. As described above, this adjustment process is repeated until the predetermined condition is satisfied, but there is a limit to the number of times the adjustment process can be repeated. Therefore, if the predetermined condition is not satisfied even after the limit is reached and automatic improvement of the response accuracy is no longer possible, the notification unit 136A notifies a predetermined notification destination that automatic improvement of the response accuracy is no longer possible. For example, the notification unit 136A may notify the administrator device 30 of the administrator T of the impossibility of improvement information.

[0073] For example, if all of the auxiliary prompts have been selected from the group of candidate auxiliary prompts without the specified conditions being met, the notification unit 136A may notify the administrator T that it is not possible to improve the response accuracy.

[0074] In addition, if all of the other multiple prepared words have been used for changes without the specified condition being met, the notification unit 136A may notify the administrator that it is impossible to improve the response accuracy.

[0075] (Identification unit 137) The identification unit 137 identifies the type of input information based on the result of intent interpretation of the input information. For example, when input information of a user U is acquired, the identification unit 137 identifies the type of input information (in this example, directions) according to the intent interpretation result.

[0076] (Prompt Adding Unit 138) The prompt adding unit 138 adds an assistance prompt to input information according to the type of the input information. For example, the prompt adding unit 138 adds an assistance prompt to input information of the user U.

[0077] [Chatbot Device 200A] As shown in FIG. 2, the chatbot device 200A according to the first embodiment includes a communication unit 210, a storage unit 220A, and a control unit 230.

[0078] (Communication Unit 210) The communication unit 210 is realized by, for example, a NIC etc. The communication unit 210 is connected to the network N by wire or wirelessly, and transmits and receives information between, for example, the user device 10, the administrator device 30, and the chatbot device 200.

[0079] (Storage unit 220A) The storage unit 220A is realized by, for example, a semiconductor memory element such as a RAM, a ROM, or a flash memory, or a storage device such as a hard disk, an SSD, or an optical disk. The storage unit 220A may store, for example, various data related to the information processing according to the first embodiment and input information.

[0080] The storage unit 220A may also store a language model. Figure 2 shows an example in which the storage unit 220A stores only one language model LLM1. In this way, in the first embodiment, the chatbot device 200A may be provided with only one fixed language model.

[0081] (Control unit 230) The control unit 230 is realized by a CPU, an MPU, etc., executing various programs stored in a storage device inside the chatbot device 200A using RAM as a work area. The control unit 230 is also realized by an integrated circuit such as an ASIC or FPGA, for example.

[0082] As shown in Fig. 2, the control unit 230 has a receiving unit 231, a response information generating unit 232, and a transmitting unit 233, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 230 is not limited to the configuration shown in Fig. 2, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units of the control unit 230 are not limited to the connection relationships shown in Fig. 2, and may be other connection relationships.

[0083] (Reception unit 231) The reception unit 231 receives various information in the information processing according to the embodiment. For example, the reception unit 231 receives input information including an assistance prompt. The reception unit 231 may also receive designation of a language model to be used to generate response information.

[0084] (Response information generation unit 232) When input information is received, the response information generation unit 232 generates response information in response to the received input information. A specified language model from among the language models stored in the storage unit 220A is used to generate the response information. For example, the response information generation unit 232 applies the language model to the input information and generates the response information based on the output result of the language model. In this way, the response information generation unit 232 can be said to be a processing unit that essentially realizes the functions of a chatbot.

[0085] (Transmission unit 233) The transmission unit 233 transmits the response information generated by the response information generation unit 232. If the input information has been received from the user U, the transmission unit 233 transmits the response information generated in response to this input information to the user device 10. As a result, a dialogue is established between the user and the chatbot. On the other hand, if the input information is verification input information input by the adjustment device 100, the transmission unit 233 transmits the response information generated in response to this input information to the adjustment device 100.

[0086] 2. Overall Processing Procedure of Information Processing According to the First Embodiment FIG. 3 is a diagram showing the overall flow of the information processing procedure according to the first embodiment, which is realized by the adjustment device 100A. The information processing according to the first embodiment includes an evaluation process for evaluating the response accuracy of the chatbot device 200A and an adjustment process for adjusting the assistance prompt. These processes are repeated until the response accuracy of the chatbot device 200A satisfies the conditions. Therefore, first, the flow of the first round of information processing according to the first embodiment will be described, and then the flow of the second round and subsequent rounds of information processing according to the first embodiment will be described.

[0087] 3 illustrates a situation in which the assistance prompt is adjusted during a verification experiment to evaluate the response accuracy of the chatbot device 200A. In such a verification experiment, the determination information generator 133 may generate verification input information at specific intervals (e.g., daily).

[0088] On the other hand, the evaluation of the response accuracy of the chatbot device 200A and the adjustment of the assistance prompts may be performed in an actual situation where the chatbot device 200A is used by the user U. In such an example, input information actually input by the user U in the actual situation may be used in the evaluation process and the adjustment process, instead of the verification input information generated by the determination information generation unit 133.

[0089] In addition, there are various types of input information that the chatbot device 200A can handle, such as "directions," "cooking recipes," and "music content," but Figure 3 shows a scene in which the chatbot device 200A is automatically adjusted in the field of "directions."

[0090] (First round of processing) The determination information generation unit 133 determines whether it is time to execute an evaluation process to evaluate the response accuracy of the chatbot device 200A (for example, once a day at 4:00 PM) (Step S101). If it is not time to execute the evaluation process (Step S101; No), the determination information generation unit 133 waits until it is time to execute the evaluation process.

[0091] On the other hand, when it is time to execute the evaluation process (step S101; Yes), the determination information generator 133 generates input information for verification based on the fixed scenario (step S102). In the example of FIG. 3 where the chatbot device 200A is automatically adjusted in the "directions" field, the determination information generator 133 uses the fixed scenario "I want to go to XX" to generate input information such as "I want to go to Tokyo Tower." Note that the determination information generator 133 may generate input information with the same content each time.

[0092] Next, the prompt assigning unit 138 determines whether the adjustment process has not yet been performed (step S103). In the first iteration of the process, the prompt assigning unit 138 determines that the adjustment process has not yet been performed (step S103; not performed) and initializes (initializes) an assistance prompt for the input information generated in step S102 (step S104). For example, the prompt assigning unit 138 may select one of the assistance prompts from the group of candidate assistance prompts stored in the prompt information storage unit 122. The candidate assistance prompts may be prepared in advance for each type of input information that the chatbot device 200A can handle. Therefore, in the example of FIG. 3, the prompt assigning unit 138 may select one of the candidate assistance prompts corresponding to "directions." For example, the prompt assigning unit 138 selects the assistance prompt PR1, "Please respond as the driver's assistant," and assigns it to the input information generated in step S102.

[0093] The sending unit 132 sends the input information IN1 including the assistance prompt PR1 to the chatbot device 200A (step S105). The chatbot device 200A applies the language model LLM1 to the input information IN1 and generates response information AN1 based on the output result of the language model LLM1.

[0094] Although the input information may be the same each time it is generated, the chatbot device 200A does not necessarily generate the same response information AN1 every time, even if the same input information is input. For example, the chatbot device 200A may generate response information that is in line with the user's intention to ask for directions, or it may generate response information AN1 that is not intended to provide directions. In other words, the content of the response information may vary.

[0095] The acquisition unit 131 acquires response information AN1 generated by the chatbot device 200A in response to the input information IN1 (step S106).

[0096] The determination unit 134A determines whether a predetermined number of pieces of response information AN1 generated by the chatbot device 200A in response to the input information IN1 have been accumulated (for example, 100 pieces of response information AN1) (step S107). If the determination unit 134A determines that the predetermined number of pieces of response information AN1 have not been accumulated (step S107; No), the process returns to step S102, and is repeated until the predetermined number of pieces of response information AN1 have been accumulated.

[0097] On the other hand, if the determination unit 134A determines that a predetermined number of pieces of response information AN1 have been accumulated (Step S107; Yes), it executes a process of calculating an evaluation value of response accuracy using the predetermined number of pieces of response information AN1 (Step S108). The detailed procedure of the evaluation process performed in Step S108 is described in FIGS. 4 and 5. Here, even though each of the predetermined number of pieces of accumulated response information AN1 is a response to the same input information IN1, due to response fluctuations of the chatbot device 200A, the response accuracy may contain different responses. Therefore, the determination unit 134A evaluates the response accuracy of the chatbot device 200A based on whether the evaluation value calculated in Step S108 satisfies a predetermined condition (Step S109).

[0098] If the evaluation value of the response accuracy satisfies a predetermined condition (step S109; Yes), the prompt adjustment unit 135A registers the current assistance prompt (e.g., assistance prompt PR1) as a generation algorithm capable of realizing the chatbot device 200A with high response accuracy (step S110). That is, the prompt adjustment unit 135A registers the assistance prompt verified to have high response accuracy as the assistance prompt to be assigned to the input information of the user U by the prompt assignment unit 138. Then, the process ends.

[0099] On the other hand, if it is determined that the evaluation value of the response accuracy does not satisfy the predetermined condition (step S109; No), the prompt adjustment unit 135A executes an adjustment process to adjust the assistance prompt (step S113). The detailed procedure of the adjustment process executed in step S113 will be described with reference to FIGS. 6 and 7.

[0100] After the prompt adjustment unit 135A performs the adjustment process, the process returns to step S102, and the process moves to the second round.

[0101] (Second and subsequent rounds of processing) In the second and subsequent rounds of processing, the determination information generation unit 133 generates input information for verification based on the fixed scenario (step S102). In the second and subsequent rounds of processing, the determination information generation unit 133 may generate the same input information, "I want to go to Tokyo Tower," as in the first round of processing.

[0102] Next, the prompt adding unit 138 determines whether the adjustment process has not yet been executed (step S103). If it is the second or subsequent iteration of the process, the prompt adding unit 138 determines that the adjustment process has not yet been executed, i.e., that the adjustment process has already been executed (step S103; execute), and then acquires an assistance prompt PR2 for which the adjustment process has already been executed in step S113, and adds the acquired assistance prompt PR2 to the input information generated in step S102 (step S114).

[0103] The sending unit 132 sends the input information IN2 including the assistance prompt PR2 to the chatbot device 200A (step S115). The chatbot device 200A applies the language model LLM1 to the input information IN2 and generates response information AN2 based on the output result of the language model LLM1.

[0104] The acquisition unit 131 acquires the response information AN2 generated by the chatbot device 200A in response to the input information IN2 (step S116).

[0105] The determination unit 134A determines whether a predetermined number of pieces of response information AN2 generated by the chatbot device 200A in response to the input information IN2 have been accumulated (for example, 100 pieces of response information AN2) (step S107). If the determination unit 134A determines that the predetermined number of pieces of response information AN2 have not been accumulated (step S107; No), the process returns to step S102, and is repeated until the predetermined number of pieces of response information AN2 have been accumulated.

[0106] On the other hand, if the determination unit 134A determines that the predetermined number of pieces of response information AN2 have been accumulated (Step S107; Yes), it executes a process of calculating an evaluation value of response accuracy using the predetermined number of pieces of response information AN2 (Step S108). Here, although each of the predetermined number of pieces of accumulated response information AN2 is a response to the same input information IN2, due to response fluctuations of the chatbot device 200A, there may be responses with different content. Therefore, the determination unit 134A evaluates the response accuracy of the chatbot device 200A based on whether the evaluation value calculated in Step S108 satisfies a predetermined condition (Step S109).

[0107] If the evaluation value of the response accuracy satisfies the predetermined condition (Step S109; Yes), the prompt adjustment unit 135A registers the current assistance prompt (e.g., assistance prompt PR2) as a generation algorithm capable of realizing the chatbot device 200A with high response accuracy (Step S110). Here, the current assistance prompt (e.g., assistance prompt PR2) is newly registered, replacing the assistance prompt (e.g., assistance prompt PR1) registered in a previous process as a generation algorithm capable of realizing the chatbot device 200A with high response accuracy. In other words, the prompt adjustment unit 135A registers the assistance prompt verified to have high response accuracy as an adjusted assistance prompt to be assigned to the input information of the user U by the prompt assignment unit 138. Then, the process ends.

[0108] On the other hand, even at the current point in time after the second round of processing, there may be cases where an assistance prompt that can obtain an evaluation value that satisfies the predetermined condition has not been found. If the prompt adjustment unit 135A determines that the evaluation value of response accuracy does not satisfy the predetermined condition (step S109; No), it executes the adjustment process again to adjust the assistance prompt (step S113).

[0109] After the prompt adjustment unit 135A performs the adjustment process, the process returns to step S102 and proceeds to the next cycle.

[0110] 3. Specific Procedures for Evaluation Processing Next, specific procedures for the evaluation processing performed in step S108 of Fig. 3 will be described. There are two types of evaluation processing: one that uses the number of characters in the response information generated by the chatbot device 200A, and one that uses the similarity between the response information generated by the chatbot device 200A and the correct response information. Fig. 4 describes the former processing procedure a, and Fig. 5 describes the latter processing procedure b.

[0111] 4 is a diagram showing a specific procedure (1) of the evaluation process according to the embodiment. First, the acquisition unit 131 acquires all of the response information accumulated in a predetermined number (e.g., 100 pieces) (step S1081a). The acquired response information includes the response information AN1 and the response information AN2 shown in FIG. 3.

[0112] The determination unit 134A calculates the number of characters in each piece of response information (step S1082a). The determination unit 134A also calculates a statistical value of the number of characters among the pieces of response information as an evaluation value for evaluating the response accuracy of the chatbot device 200A (step S1083a). For example, the more helpful the response information generated by the chatbot device 200A, the more unnecessary information that deviates from the user's intention is included. Therefore, by focusing on the number of characters, the response accuracy of the chatbot device 200A can be appropriately evaluated.

[0113] The determining unit 134A may calculate, as the statistical value, at least one of the quartile, the mode, and the minimum value, for example.

[0114] Then, the judgment unit 134A may perform the evaluation process of step S109 in Figure 2, which evaluates the response accuracy of the chatbot device 200A based on whether the evaluation value satisfies a predetermined condition, by determining whether the statistical value of the number of characters calculated in S1083a satisfies a threshold condition.

[0115] Here, if the statistical value of the number of characters satisfies the threshold condition (step S109; Yes), this means that the response accuracy of the chatbot device 200A is high, and therefore the processing proceeds to step S110.

[0116] On the other hand, if the statistical value of the number of characters does not satisfy the threshold condition (step S109; No), this means that the response accuracy of the chatbot device 200A is low, and the process proceeds to step S113.

[0117] Note that, when the determination unit 134A calculates, for example, quartiles, modes, and minimum values ​​as multiple types of character count statistical values, it may perform condition determinations individually, such as whether the quartiles satisfy the threshold condition, whether the modes satisfy the threshold condition, and whether the minimum values ​​satisfy the threshold condition.The determination unit 134A may then evaluate the response accuracy based on whether a predetermined percentage or more (e.g., a majority) of the multiple types of character count statistical values ​​(e.g., quartiles and modes) each satisfy the threshold condition, or whether all of the multiple types of character count statistical values ​​(e.g., quartiles, modes, and minimum values) each satisfy the threshold condition.

[0118] Next, the process of Fig. 5 will be described. Fig. 5 is a diagram showing a specific procedure (2) of the evaluation process according to the embodiment. In the example of Fig. 5, the acquisition unit 131 also acquires all of the response information accumulated in a predetermined number (e.g., 100 pieces) (step S1081b). The acquired response information includes the response information AN1 and the response information AN2 shown in Fig. 3.

[0119] The acquiring unit 131 also acquires correct answer information (correct response information) that is prepared in advance for the input information (step S1082b).

[0120] The determination unit 134A then calculates the similarity between each piece of response information and the correct answer information (step S1083b). The determination unit 134A also calculates a statistical value of the similarity between the response information as an evaluation value for evaluating the response accuracy of the chatbot device 200A (step S1084a). For example, the determination unit 134A may calculate the cosine similarity between each piece of response information and the correct answer information. For example, the more similar the response information generated by the chatbot device 200A is to the correct answer information, the more likely it is that the response information is composed of only appropriate information that matches the user's intention. Therefore, by focusing on the similarity, the response accuracy of the chatbot device 200A can be appropriately evaluated.

[0121] Then, the judgment unit 134A may perform a process of judging whether the statistical value of the similarity calculated in S1083a satisfies a threshold condition as the evaluation process of step S109 in Figure 2, which evaluates the response accuracy of the chatbot device 200A based on whether the evaluation value satisfies a predetermined condition.

[0122] Here, if the statistical value of similarity satisfies the threshold condition (step S109; Yes), this means that the response accuracy of the chatbot device 200A is high, and therefore the processing proceeds to step S110.

[0123] On the other hand, if the similarity statistical value does not satisfy the threshold condition (step S109; No), this means that the response accuracy of the chatbot device 200A is low, and processing proceeds to step S113.

[0124] Note that, when the determination unit 134A calculates, for example, quartiles, modes, and minimum values ​​as the multiple types of similarity statistical values, it may perform condition determinations individually, such as determining whether the quartiles satisfy the threshold condition, whether the modes satisfy the threshold condition, and whether the minimum values ​​satisfy the threshold condition.The determination unit 134A may then evaluate the response accuracy based on whether a predetermined percentage or more (e.g., a majority) of the multiple types of similarity statistical values ​​(e.g., quartiles and modes) each satisfy the threshold condition, or whether all of the multiple types of similarity statistical values ​​(e.g., quartiles, modes, and minimum values) each satisfy the threshold condition.

[0125] 3 and 4 show the evaluation process procedure as a method for evaluating the response accuracy of the chatbot device 200A according to the first embodiment, but the response accuracy of the chatbot device 200B according to the second embodiment is also evaluated in a similar manner. Therefore, in the second embodiment described below, the specific procedure for the evaluation process for the chatbot device 200B will not be described.

[0126] 4. Specific Procedures for Adjustment Processing Next, specific procedures for the adjustment processing performed in step S113 of Fig. 3 will be described. The adjustment processing for adjusting the assist prompt is classified into a process for changing the entire assist prompt and a process for changing only some of the words included in the assist prompt. Fig. 6 describes the former process step a, and Fig. 7 describes the latter process step b.

[0127] 6 is a diagram showing a specific procedure (1) of the adjustment process according to the first embodiment. The prompt adjustment unit 135A determines whether there are any unselected assistance prompts among the prepared assistance prompts (step S1131a). For example, the prompt adjustment unit 135A may specify a change candidate assistance prompt group corresponding to the type "directions" from among the change candidate assistance prompt groups prepared in advance for each type of input information that the chatbot device 200A can handle. The prompt adjustment unit 135A may then determine whether there are any unselected assistance prompts among the change candidate assistance prompt group corresponding to the type "directions."

[0128] If there are unselected help prompts among the prepared help prompts (step S1131a; Yes), the prompt adjustment unit 135A selects one of the unselected help prompts (step S1132a). For example, suppose that the help prompt PR1, "Please respond as the driver's assistant," is currently being adjusted, and that there are unselected help prompts PRC11 and PRC12, as shown in FIG. 6. In this example, the prompt adjustment unit 135A can select one of the help prompts PRC11 and PRC12.

[0129] The prompt adjustment unit 135A then replaces the current assistance prompt to be adjusted with the assistance prompt selected in step S1132a (step S1133a).

[0130] Next, the prompt adjustment unit 135A stores the replaced assistance prompt as an assistance prompt for which adjustment processing has been performed (step S1134a). Then, the process proceeds to step S101. For example, assume that the assistance prompt PR1, "Please respond as the driver's assistant," is replaced with the candidate assistance prompt, "Please guide to a destination that meets the driver's wishes." In this example, the assistance prompt, "Please guide to a destination that meets the driver's wishes," is stored as an assistance prompt PR2 for which adjustment processing has been performed and is acquired in step S114 of FIG. 3 .

[0131] On the other hand, if there are no unselected auxiliary prompts among the group of pre-prepared auxiliary prompts (step S1131a; No), that is, if all the pre-prepared auxiliary prompts have been used up, the prompt adjustment unit 135A recognizes that no response information was generated that would result in an evaluation value that satisfied the conditions no matter which auxiliary prompt was given to the chatbot device 200A (no use of any auxiliary prompt would have improved the response accuracy of the chatbot device 200A) (step S1135a).

[0132] Then, the notification unit 136A notifies the administrator T that the response accuracy of the chatbot device 200A cannot be improved (step 1136a).

[0133] Next, the process shown in FIG. 7 will be described. FIG. 7 is a diagram showing a specific procedure (2) of the adjustment process according to the embodiment. The prompt adjustment unit 135A extracts role-designating words from among the words included in the assistance prompt currently being adjusted (step 1131b). Role-designating words refer to words that designate roles according to the type of input information that the chatbot device 200A can handle. For example, role-designating words corresponding to the type "directions" include "assistant," "butler," "passenger," and "driving partner." Although "assistant," "butler," "passenger," and "driving partner" are different expressions, they are synonymous with each other.

[0134] The prompt adjuster 135A determines whether or not there is an unselected role-specifying word in the prepared role-specifying word group (step S1132b).

[0135] If there are unselected role-designating words in the prepared group of role-designating words (step S1132b; Yes), the prompt adjustment unit 135A selects one of the unselected role-designating words (step S1133b). For example, suppose that the current adjustment target is the assistance prompt PR1, "Please respond as the driver's assistant," and that the unselected role-designating words are the role-designating words WDC21, WDC22, and WDC23, as shown in FIG. 7. In this example, the prompt adjustment unit 135A can select one of the role-designating words WDC21 to WDC23.

[0136] The role-designating words may be prepared in advance by the administrator T and registered in the prompt information storage unit 122. On the other hand, the role-designating words may be listed by the chatbot device 200. For example, the administrator T may request the chatbot device 200 to present role words by issuing a command such as "Please think of a word that represents the role of assisting the driver."

[0137] Returning to the description of FIG. 7, the prompt adjustment unit 135A changes the role-specifying word currently included in the assistance prompt to be adjusted to the role-specifying word selected in step S1133b (step S1134b).

[0138] The prompt adjustment unit 135A also replaces the assistance prompt currently being adjusted with the assistance prompt after the role specifying word has been changed (step S1135b).

[0139] Next, the prompt adjustment unit 135A stores the replaced assistance prompt as an assistance prompt for which adjustment processing has been performed (step S1136b). Then, the process proceeds to step S101. For example, suppose the role-designating word "assistant" included in assistance prompt PR1, "Please respond as the driver's assistant," is changed to another role-designating word, "butler." In this example, the assistance prompt "Please respond as the driver's butler" is stored as assistance prompt PR2 for which adjustment processing has been performed, and is acquired in step S114 of FIG. 3.

[0140] On the other hand, if there are no unselected role-designating words in the group of pre-prepared role-designating words (step S1132b; No), that is, if all the pre-prepared role-designating words have been used up, the prompt adjustment unit 135A recognizes that no response information will be generated that will result in an evaluation value that satisfies the conditions, even if an auxiliary prompt containing any role-designating word is given to the chatbot device 200A (using an auxiliary prompt containing any role-designating word did not improve the response accuracy of the chatbot device 200A) (step S1137b).

[0141] Then, the notification unit 136A notifies the administrator T that the response accuracy of the chatbot device 200A cannot be improved (step 1138b).

[0142] 5. Dialogue Control Process Using Adjusted Assistance Prompts Assume that, through the adjustment process described above, the adjustment device 100A according to the first embodiment has been able to adjust an assistance prompt that enables the chatbot device 200A to achieve high response accuracy. In this case, the adjustment device 100A controls the chatbot device 200A to generate response information corresponding to input information from the user U in accordance with the adjusted assistance prompts. This type of dialogue control will be described with reference to FIG. 8. FIG. 8 is a diagram showing the dialogue control procedure executed by the adjustment device 100A according to the first embodiment.

[0143] First, the acquisition unit 131 determines whether or not the input information INU of the user U has been acquired (step S201).

[0144] The identification unit 137 identifies the type of the input information INU based on the result of the intention interpretation of the input information INU (step S202).

[0145] The prompt granting unit 138 acquires an auxiliary prompt corresponding to the type of input information INU from among the auxiliary prompts that can obtain an evaluation value that satisfies the conditions (i.e., auxiliary prompts that can realize a chatbot device 200A with high response accuracy) (step S203).

[0146] The prompt assigning unit 138 then assigns the assistance prompt acquired in step S203 to the input information INU (step S204). For example, assume that the type of input information INU is "directions" among the types of input information that the chatbot device 200A can handle, and that assistance prompt PR2 has been registered as an assistance prompt that can realize the chatbot device 200A with high response accuracy. In this example, the prompt assigning unit 138 acquires the assistance prompt PR2 and assigns it to the input information INU.

[0147] In addition, the transmission unit 132 transmits the input information INU including the assistance prompt to the chatbot (step S205).

[0148] In this state, the acquisition unit 131 determines whether or not it has acquired the response information ANU generated by the chatbot device 200A in response to the input information INU (step S206). If it has not acquired the response information ANU (step S206; No), the acquisition unit 131 waits until it can acquire the response information ANU.

[0149] On the other hand, if the transmitting unit 132 has acquired the response information ANU (step S206; Yes), it transmits the response information ANU to the user device 10 (step S207). Although not shown in Fig. 8, the output control unit 15b of the user device 10 performs output control to output the response information ANU from the output unit 14.

[0150] Second Embodiment Next, a second embodiment will be described. The information processing according to the first embodiment adjusts the assistance prompt in accordance with the evaluation result of the response accuracy of the chatbot device 200. In contrast, the information processing according to the second embodiment selects a chatbot to be executed in accordance with the evaluation result of the response accuracy of the chatbot device 200B, and includes an adjustment process of repeating the selection of the chatbot to be executed until the evaluation result exceeds a pass mark.

[0151] As will be described later, in the second embodiment, selecting a chatbot to be executed includes, for example, either selecting one chatbot device 200 from among chatbot devices, or selecting a language model to be executed on one chatbot device 200 from among multiple candidates.

[0152] [1. Functional Configuration] Using FIG. 9, an example configuration of the adjustment device 100B and the chatbot device 200B according to the second embodiment will be described. FIG. 9 is a diagram showing an example device configuration according to the second embodiment. In FIG. 9, explanations of points common to the first embodiment will be omitted or simplified. For example, the operation of the user device 10 is the same in both the first and second embodiments, and therefore explanations will be omitted. Furthermore, explanations of processing units that are assigned the same reference numerals as those in the first embodiment will be omitted or simplified.

[0153] [Adjustment Device 100B] As shown in FIG. 9, an adjustment device 100B according to the second embodiment includes a communication unit 110, a storage unit 120B, and a control unit 130B.

[0154] (Storage Unit 120B) The storage unit 120B is realized by, for example, a semiconductor memory element such as a RAM, a ROM, or a flash memory, or a storage device such as a hard disk, an SSD, or an optical disk. The storage unit 120B may store, for example, data and programs related to the information processing according to the second embodiment. Furthermore, according to the example of FIG. 2 , the storage unit 120B may further include a language model information storage unit 123 in addition to the correct answer information storage unit 121 and the prompt information storage unit 122.

[0155] (Language Model Information Storage Unit 123) In the information processing according to the second embodiment, an arbitrary language model is selected from a group of language models as selection candidates, and the selected language model is specified for the chatbot device 200B. As a result, the chatbot device 200B generates response information using, for example, the specified language model among the language models registered in the storage unit 220B. Therefore, the language model information storage unit 123 may store information about the language models as selection candidates, and the stored language models as selection candidates correspond to the language models registered in the storage unit 220B as shown in FIG. 9 . For example, the information about the language models as selection candidates may be registered in advance by the administrator T. Note that the language model information storage unit 123 may store an access destination address associated with each language model when access to a physically different chatbot device 200B is required for each language model used.

[0156] (Control unit 130B) The control unit 130B is realized by a CPU, an MPU, etc., executing various programs (e.g., the information processing program according to the second embodiment) stored in a storage device inside the adjustment device 100B using RAM as a work area. The control unit 130B is also realized by an integrated circuit such as an ASIC or FPGA.

[0157] As shown in Fig. 9, the control unit 130B has an acquisition unit 131, a transmission unit 132, a determination information generation unit 133, a determination unit 134B, an execution target adjustment unit 135B, a notification unit 136B, an identification unit 137, and a prompt assignment unit 138, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130B is not limited to the configuration shown in Fig. 9, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units of the control unit 130A are not limited to the connection relationships shown in Fig. 2, and may be other connection relationships.

[0158] (Determination unit 134B) The determination unit 134B determines (evaluates) the response accuracy of the chatbot device 200B based on whether or not the response information generated by the chatbot device 200B in response to input information satisfies a predetermined condition. Specifically, the determination unit 134B determines whether or not the first response information generated by the first chatbot device 200B in response to input information satisfies a predetermined condition.

[0159] If the first response information does not satisfy the predetermined conditions and the response accuracy of the first chatbot device 200B is determined to be low, an adjustment process is executed to select a second chatbot device 200B having a language model different from that of the first chatbot device 200B as an execution target for generating response information. Therefore, the determination unit 134B determines whether the second response information generated by the second chatbot device 200B in response to the input information satisfies the predetermined conditions.

[0160] If the second response information does not satisfy the predetermined condition and the response accuracy of the second chatbot device 200B is determined to be low, an adjustment process is further executed to select a third chatbot device 200B having a language model different from the first chatbot device 200B and the second chatbot device 200B as an execution target for generating response information. Therefore, the determination unit 134B further determines whether the third response information generated by the third chatbot device 200B in response to the input information satisfies the predetermined condition.

[0161] In this way, in the information processing according to the second embodiment, the selection of an execution target is repeated until the condition that the response accuracy of the chatbot device 200B exceeds the pass mark is met. Therefore, it is possible that the selection is repeated in a manner not limited to the above example, such that the fourth chatbot device 200B is selected as the execution target for generating response information, and then the fifth chatbot device 200B is selected as the execution target for generating response information.

[0162] For example, the determination unit 134B calculates the number of characters in each piece of first response information generated by the first chatbot device 200B each time the same piece of input information is input. Then, based on the calculated number of characters, the determination unit 134B calculates a statistical value of the number of characters among a predetermined number of pieces of first response information, and determines whether the statistical value satisfies a predetermined condition.

[0163] As another example, each time the same input information is input, the determination unit 134B calculates the similarity between each piece of first response information generated by the first chatbot device 200B and the correct answer information prepared in advance for the input information. Then, based on the calculated similarity, the determination unit 134B calculates a statistical value of the similarity between a predetermined number of pieces of first response information and determines whether the statistical value satisfies a predetermined condition.

[0164] The determination unit 134B may calculate quartiles, modes, minimum values, etc. as the multiple types of statistical values, and determine whether or not a predetermined number of the multiple types of statistical values ​​satisfy a predetermined condition. For example, a corresponding condition may be set for each statistical value, such as the quartiles, modes, and minimum values, and the determination unit 134B may evaluate the response accuracy based on whether or not a predetermined percentage or more of the multiple types of statistical values ​​satisfy the condition, or whether all of the multiple types of statistical values ​​satisfy the condition.

[0165] Here, the above example shows an example in which the judgment unit 134B evaluates the response accuracy of the first chatbot device 200B, but the judgment unit 134B also evaluates the response accuracy of the second chatbot device 200B, the third chatbot device 200B, etc. using a similar method.

[0166] (Execution Target Adjustment Unit 135B) When the execution target adjustment unit 135B determines that the first chatbot device 200B's response accuracy is low because the first response information generated by the first chatbot device 200B in response to the input information does not satisfy a predetermined condition, the execution target adjustment unit 135B selects a second chatbot device 200B having a language model different from that of the first chatbot device 200B as the execution target for generating response information. When the execution target adjustment unit 135B determines that the second response information generated by the second chatbot device 200B in response to the input information does not satisfy a predetermined condition and the response accuracy of the second chatbot device 200B is low, the execution target adjustment unit 135B selects a third chatbot device 200B having a language model different from that of the first chatbot device 200B and the second chatbot device 200B as the execution target for generating response information.

[0167] In this way, the adjustment process of selecting an execution target from among the chatbot devices 200B to generate response information is repeated until the condition that the response accuracy of the execution target chatbot device 200B exceeds the pass mark is met. Also, selecting an execution target corresponds to selecting an arbitrary language model from a group of language models of selection candidates prepared in advance, and replacing the language model installed as a function of the currently execution target chatbot device 200B with the selected language model.

[0168] The process of selecting an execution target for generating response information will be described in detail below. The execution target adjustment unit 135B selects an arbitrary language model from a group of language models of selection candidates prepared in advance, and replaces the language model installed as a function of the chatbot currently being executed with the selected language model. More specifically, if a predetermined condition is not satisfied, the execution target adjustment unit 135B repeats the replacement process of selecting an arbitrary language model from the group of language models of selection candidates and replacing the language model installed as a function of the chatbot currently being executed with the selected language model until the predetermined condition is satisfied.

[0169] In addition, if the execution target adjustment unit 135B requires access to a physically different chatbot device 200B for each language model selected as an execution target for generating response information, the transmission unit 132 may refer to the access destination address for each language model stored in the language model information storage unit 123 and send the input information IN to that address, and the acquisition unit 131 may acquire the response information generated by the chatbot device 200B in response to the input information IN.

[0170] (Notification unit 136B) According to the description above, if the response information generated by the chatbot device 200B in response to input information does not satisfy a predetermined condition and the response accuracy of the chatbot device 200B is determined to be low, an adjustment process is executed to automatically improve the response accuracy. As described above, this adjustment process is repeated until the predetermined condition is satisfied, but there is a limit to the number of times the adjustment process can be repeated. Therefore, if the predetermined condition is not satisfied even after the limit is reached and automatic improvement of the response accuracy is no longer possible, the notification unit 136B notifies a predetermined notification destination that automatic improvement of the response accuracy is no longer possible. For example, the notification unit 136B may notify the administrator device 30 of the administrator T of the impossibility of improvement information.

[0171] For example, if all language models have been selected from the group of language models as selection candidates without the specified conditions being met, the notification unit 136B may notify the administrator T that it is not possible to improve the response accuracy.

[0172] 2. Overall Processing Procedure of Information Processing According to the Second Embodiment FIG. 10 is a diagram showing the overall flow of the information processing procedure according to the second embodiment, which is realized by the adjustment device 100B. The information processing according to the second embodiment includes an evaluation process for evaluating the response accuracy of the chatbot device 200B and an adjustment process for selecting an execution target for generating response information. These processes are repeated until the response accuracy of the chatbot device 200B satisfies the conditions. Therefore, first, the flow of the first round of information processing according to the second embodiment will be described, and then the flow of the second round and subsequent rounds of information processing according to the second embodiment will be described.

[0173] 10 shows a scene in which an execution target is selected in a verification experiment to evaluate the response accuracy of the chatbot device 200B. In such a verification experiment, the determination information generator 133 may generate verification input information at specific times (e.g., once a day).

[0174] On the other hand, the evaluation of the response accuracy of the chatbot device 200B and the selection of an execution target may be performed in an actual situation where the chatbot device 200B is used by the user U. In such an example, input information actually input by the user U in the actual situation may be used in the evaluation process and the adjustment process, instead of the input information for verification generated by the determination information generation unit 133.

[0175] Furthermore, there are various types of input information that the chatbot device 200B can handle, such as "directions," "cooking recipes," and "music content," but Figure 3 shows a scene in which the chatbot device 200B is automatically adjusted in the field of "directions."

[0176] (First round of processing) The determination information generation unit 133 determines whether it is time to execute an evaluation process to evaluate the response accuracy of the chatbot device 200B (for example, once a day at 4:00 PM) (Step S301). If it is not time to execute the evaluation process (Step S301; No), the determination information generation unit 133 waits until it is time to execute the evaluation process.

[0177] On the other hand, when it is time to execute the evaluation process (step S301; Yes), the determination information generator 133 generates input information for verification based on the fixed scenario (step S302). In the example of FIG. 10 where the chatbot device 200B is automatically adjusted in the "directions" field, the determination information generator 133 uses the fixed scenario "I want to go to XX" to generate input information IN such as "I want to go to Tokyo Tower." Note that the determination information generator 133 may generate input information with the same content each time.

[0178] Next, the execution target adjustment unit 135B determines whether or not the adjustment process has not yet been performed (step S303). In the case of the first round of processing, the execution target adjustment unit 135B determines that the adjustment process has not yet been performed (step S303; not performed) and initially specifies a language model (step S304). For example, the execution target adjustment unit 135B may select one arbitrary language model from the group of language models that are candidates for modification stored in the language model information storage unit 123, and initially specify the selected language model. In the example of FIG. 10, the execution target adjustment unit 135B selects the language model LLM1 and initially specifies the language model LLM1.

[0179] The transmitter 132 transmits information to the chatbot device 200B that sets the language model LLM1 specified by the execution target adjuster 135B as the language model to be used in subsequent dialogues. Alternatively, the transmitter 132 may transmit information that sets the language model LLM1 as the dialogue language model prior to the input information IN each time the input information IN is transmitted to the chatbot device 200B. The transmitter 132 then transmits the input information IN to the chatbot CB1 (an example of the first chatbot device 200B) that is equipped with the language model LLM1 (step S305). The chatbot CB1 applies the language model LLM1 to the input information IN and generates response information AN1 based on the output result of the language model LLM1.

[0180] An assistance prompt may be added to the input information IN sent to the chatbot CB1. For example, the prompt adding unit 138 may select one of the assistance prompts stored in the prompt information storage unit 122 as candidate changes, and add the selected assistance prompt to the input information IN generated in step S302.

[0181] In the information processing according to the second embodiment, the assistance prompts provided in each round may be fixed. On the other hand, in the information processing according to the second embodiment, different assistance prompts may be provided in each round to search for an optimal combination (combination of assistance prompt and language model) that will result in a high response accuracy of the chatbot device 200. In other words, the information processing according to the first embodiment may be performed in parallel with the information processing according to the second embodiment.

[0182] Returning to the explanation of FIG. 10, the acquisition unit 131 acquires response information AN1 generated by the chatbot CB1 in response to the input information IN (step S306).

[0183] The determination unit 134B determines whether a predetermined number of pieces of response information AN1 generated by the chatbot CB1 in response to the input information IN have been accumulated (for example, 100 pieces have been accumulated) (step S307). If the determination unit 134B determines that the predetermined number of pieces of response information AN1 have not been accumulated (step S307; No), the process returns to step S302, and is repeated until the predetermined number of pieces of response information AN1 have been accumulated.

[0184] On the other hand, if the determination unit 134B determines that a predetermined number of pieces of response information AN1 have been accumulated (step S307; Yes), it executes a process of calculating an evaluation value of response accuracy using the predetermined number of pieces of response information AN1 (step S308). The evaluation process performed in step S308 is the same as the pattern of the first embodiment described with reference to FIGS. 4 and 5, and therefore will not be described again. Here, although each of the predetermined number of pieces of accumulated response information AN1 is a response to the same input information IN, due to response fluctuations of the chatbot CB1, there may be responses with different content. Therefore, the determination unit 134B evaluates the response accuracy of the chatbot CB1 based on whether the evaluation value calculated in step S308 satisfies a predetermined condition (step S309).

[0185] If the evaluation value of response accuracy satisfies the predetermined condition (step S309; ​​Yes), the execution target adjustment unit 135B registers the language model currently being executed (e.g., language model LLM1) as a generation algorithm capable of realizing the chatbot device 200B with high response accuracy (step S310). In other words, the execution target adjustment unit 135B registers the chatbot having the language model verified to have high response accuracy as the execution target for generating response information. Then, the processing ends.

[0186] On the other hand, if it is determined that the evaluation value of the response accuracy does not satisfy the predetermined condition (step S309; ​​No), the execution target adjustment unit 135B executes an adjustment process to select a language model to be executed (step S313). The detailed procedure of the adjustment process executed in step S313 will be described with reference to FIG. 11.

[0187] After the adjustment process is performed by the execution target adjustment unit 135B, the process returns to step S302, and the process moves to the second round of processing.

[0188] (Second and subsequent rounds of processing) In the second and subsequent rounds of processing, the determination information generation unit 133 generates input information for verification based on the fixed scenario (step S302). In the second and subsequent rounds of processing, the determination information generation unit 133 may generate the same input information, "I want to go to Tokyo Tower," as in the first round of processing.

[0189] Next, the execution target adjustment unit 135B determines whether the adjustment process has not yet been executed (step S303). If it is the second or subsequent round of processing, the execution target adjustment unit 135B determines that the adjustment process has not yet been executed, i.e., that the adjustment process has been executed (step S303; execute), and specifies the language model selected as the execution target by the adjustment process in step S313 (step S314). Here, it is assumed that the language model LLM2 was selected in the adjustment process in step S313.

[0190] The sending unit 132 sends the input information IN to the chatbot CB2 (an example of the second chatbot device 200B) equipped with the language model LLM2 (step S315). As described above, the input information IN sent to the chatbot CB2 may be provided with an assistance prompt of the same content as that in the first processing round, or may be provided with an assistance prompt of a different content.

[0191] The chatbot CB2 applies the language model LLM2 to the input information IN and generates response information AN2 based on the output result of the language model LLM2.

[0192] The acquisition unit 131 acquires response information AN2 generated by the chatbot CB2 in response to the input information IN (step S316).

[0193] The determination unit 134B determines whether a predetermined number of pieces of response information AN2 generated by the chatbot CB2 in response to the input information IN have been accumulated (for example, 100 pieces have been accumulated) (step S307). If the determination unit 134B determines that the predetermined number of pieces of response information AN2 have not been accumulated (step S307; No), the process returns to step S302, and is repeated until the predetermined number of pieces of response information AN2 have been accumulated.

[0194] On the other hand, if the determination unit 134A determines that the predetermined number of pieces of response information AN2 have been accumulated (step S307; Yes), it executes a process of calculating an evaluation value of response accuracy using the predetermined number of pieces of response information AN2 (step S308). Here, although each of the predetermined number of pieces of accumulated response information AN2 is a response to the same input information IN, there may be cases where the response content differs due to response fluctuations of the chatbot CB2. Therefore, the determination unit 134B evaluates the response accuracy of the chatbot CB2 based on whether the evaluation value calculated in step S308 satisfies a predetermined condition (step S309).

[0195] If the evaluation value of response accuracy satisfies the predetermined condition (step S309; ​​Yes), the execution target adjustment unit 135B registers the currently executed language model (e.g., language model LLM2) as a generation algorithm capable of realizing the chatbot device 200B with high response accuracy (step S310). Here, the current language model (e.g., language model LLM2) is newly registered, replacing the language model (e.g., language model LLM1) registered in a previous process as a generation algorithm capable of realizing the chatbot device 200B with high response accuracy. In other words, the execution target adjustment unit 135B registers the chatbot having the language model verified to have high response accuracy as an execution target for generating response information. The process then ends.

[0196] On the other hand, even at the current point in time after the second round of processing, there may be cases where a language model that can obtain an evaluation value that satisfies the predetermined condition has not been found. If it is determined that the evaluation value of response accuracy does not satisfy the predetermined condition (Step S309; ​​No), the execution target adjustment unit 135B executes the adjustment process of selecting a language model again (Step S313).

[0197] After the adjustment process is performed by the execution target adjustment unit 135B, the process returns to step S302, and the process proceeds to the next cycle.

[0198] 3. Specific Procedure of Adjustment Processing Next, a specific procedure of the adjustment processing (adjustment processing for selecting a language model to be executed) performed in step S313 of FIG. 10 will be described.

[0199] 11 is a diagram showing a specific procedure of the adjustment process according to the second embodiment. The execution target adjustment unit 135B determines whether or not there is an unselected language model in the group of language models prepared in advance (step S3131).

[0200] If there is an unselected language model in the group of language models prepared in advance (step S3131; Yes), the execution target adjustment unit 135B selects one of the unselected language models (step S3132). For example, assume that the language model LLM1 is currently the execution target, and that the unselected language models are language models MC31 and MC32, as shown in Figure 11. In this example, the execution target adjustment unit 135B can select one of the language models MC31 and MC32.

[0201] Then, the execution target adjustment unit 135B replaces the language model currently being executed with the language model selected in step S3132 (step S3133).

[0202] Next, the execution target adjustment unit 135B retains the replaced language model as the language model selected as the execution target by the adjustment process (step S3134). Then, the process proceeds to step S301. For example, assume that the language model LLM1 is replaced with the candidate language model LLM2. In this example, the language model LLM2 is retained as the language model selected as the execution target by the adjustment process and is specified in step S314 of FIG. 10.

[0203] On the other hand, if there are no unselected language models among the group of pre-prepared language models (step S3131; No), that is, if all pre-prepared language models have been used up, the execution target adjustment unit 135B recognizes that no matter which language model is installed in the chatbot device 200B, response information will not be generated so as to obtain an evaluation value that satisfies the conditions (no matter which language model is used, the response accuracy of the chatbot device 200B will not improve) (step S3135).

[0204] Then, the notification unit 136B notifies the administrator T that the response accuracy of the chatbot device 200B cannot be improved (step 3136).

[0205] Other Embodiments The information processing according to each of the above-described embodiments is not limited to application in a pull-type system such as a chatbot, but can also be extended to a push-type system that automatically makes announcements to users. For example, a case where the information processing according to each embodiment is applied to an automatic announcement system that takes the surrounding environment into consideration will be described. In such a case, the adjustment device 100 acquires the surrounding traffic congestion status as a parameter in conjunction with the current location, and converts the acquired parameter into a prompt and inputs it to the chatbot device 200. The chatbot device 200 may generate response information for various driving assistance depending on the traffic congestion status.

[0206] For this reason, by extending the information processing according to each embodiment to a push-type automatic announcement system, it becomes possible to realize a system that does not require the user to input a prompt, and that spontaneously provides user assistance, including issuing warnings according to the situation.

[0207] Even when the information processing according to each embodiment is extended to a push-type automatic announcement system, for example, the response information may be helpful or may contain words unrelated to the purpose of the response. In this way, even when it is determined that the response accuracy of the chatbot device 200 has decreased in a push-type service, it is possible to automatically improve the response accuracy.

[0208] <Hardware Configuration> The above-described adjustment device 100 may be realized, for example, by a computer 1000 configured as shown in Fig. 12. Fig. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the adjustment device 100 according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0209] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0210] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0211] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. The CPU 1100 also outputs generated data to the output device via the input / output interface 1600.

[0212] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0213] For example, when the computer 1000 functions as the adjustment device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0214] <Others> Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0215] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0216] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0217] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art.

[0218] REFERENCE SIGNS LIST 1 System 10 User device 30 Administrator device 100A Adjustment device 134A Determination unit 135A Prompt adjustment unit 136A Notification unit 100B Adjustment device 134B Determination unit 135B Execution target adjustment unit 136B Notification unit 200 Chatbot device

Claims

1. A determination unit that acquires response information generated by a chatbot according to input information and determines the response accuracy of the chatbot based on whether the acquired response information meets a predetermined condition; When it is determined that the response information does not meet the predetermined condition and the response accuracy of the chatbot is low, an adjustment unit that executes a predetermined process for automatically improving the response accuracy; A notification unit that, when the response information acquired by the determination unit does not meet the predetermined condition after the adjustment unit executes the predetermined process, notifies a predetermined notification destination that the automatic improvement of the response accuracy is impossible; An information processing apparatus characterized by comprising.

2. The determination unit determines whether first response information generated by the chatbot for the first input information meets the predetermined condition, When it is determined that the first response information does not meet the predetermined condition and the response accuracy of the chatbot is low, the adjustment unit executes a process of adjusting an auxiliary prompt that is at least a part of the instruction text included in the first input information. The information processing apparatus according to claim 1.

3. The determination unit determines whether second response information generated by the chatbot for second input information including the adjusted auxiliary prompt as at least a part of the instruction text meets the predetermined condition, When it is determined that the second response information does not meet the predetermined condition and the response accuracy of the chatbot is low, the adjustment unit executes a process of readjusting the adjusted auxiliary prompt. The information processing apparatus according to claim 1.

4. The determination unit determines whether first response information generated by a first chatbot for input information meets a predetermined condition, When it is determined that the first response information does not meet the predetermined condition and the response accuracy of the first chatbot is low, the adjustment unit executes a process of selecting a second chatbot having a language model different from that of the first chatbot as an execution target for generating response information. The information processing apparatus according to claim 1.

5. The determination unit determines whether second response information generated by the second chatbot for the input information satisfies the predetermined condition, and when it is determined that the second response information does not satisfy the predetermined condition and the response accuracy of the second chatbot is low, the adjustment unit selects, as the execution target, a third chatbot having a language model different from the first chatbot and the second chatbot, and executes a process of selecting the third chatbot. The information processing apparatus according to claim 4.

6. The determination unit acquires the response information generated by the chatbot each time the input information of the same content is input, calculates a statistical value of the number of characters among a predetermined number of the response information based on the number of characters of each of the acquired response information, and determines whether the statistical value satisfies the predetermined condition. The information processing apparatus according to claim 1.

7. The determination unit acquires the response information generated by the chatbot each time the input information of the same content is input, calculates a statistical value of the similarity among a predetermined number of the response information based on the similarity between each of the acquired response information and correct answer information prepared in advance for the input information, and determines whether the statistical value satisfies the predetermined condition. The information processing apparatus according to claim 1.

8. The determination unit calculates a plurality of types of the statistical values, and determines whether more than a predetermined number of the statistical values among the plurality of types of the statistical values satisfy the predetermined condition. The information processing apparatus according to claim 6 or 7.

9. An information processing method executed by an information processing apparatus, including: a determination step of acquiring response information generated by a chatbot according to input information, and determining the response accuracy of the chatbot based on whether the acquired response information satisfies a predetermined condition; an adjustment step of executing a predetermined process for automatically improving the response accuracy when it is determined that the response information does not satisfy the predetermined condition and the response accuracy of the chatbot is low; and a notification step of notifying a predetermined notification destination that the automatic improvement of the response accuracy is impossible when the response information acquired in the determination step does not satisfy the predetermined condition after the adjustment step executes the predetermined process. The information processing method including the above steps.

10. An information processing program executed by an information processing apparatus, comprising: a determination procedure for obtaining response information generated by a chatbot according to input information and determining the response accuracy of the chatbot based on whether the obtained response information meets a predetermined condition; an adjustment procedure for executing a predetermined process for automatically improving the response accuracy when it is determined that the response information does not meet the predetermined condition and the response accuracy of the chatbot is low; and a notification procedure for notifying a predetermined notification destination that the automatic improvement of the response accuracy is impossible when the response information obtained by the determination procedure does not meet the predetermined condition after the adjustment procedure executes the predetermined process. An information processing program for causing the information processing apparatus to execute the above.

Images