Control device, robot system, control method. and control program

Abstract

To provide a control device, control method and control program that enables natural conversation with a user.SOLUTION: In a robot system 100, a control device includes: an audio input unit that receives voice input corresponding to speech from a user; an audio output unit that outputs voice corresponding to a response to the speech to the user; an audio data storage unit that pre-stores audio data of a predetermined number of fixed phrases used at beginning of responses; a processing unit that sends speech data to a natural language processing system and receives a response; and a voice synthesis unit that performs voice synthesis based on the response. The processing unit includes a classifier that predicts an initial fixed phrases for the response, and reads the audio data corresponding to the predicted initial response from the audio data storage unit, and outputs audio to the audio output unit based on the audio data. The voice synthesis unit synthesizes the voice corresponding to the response received from the natural language processing system, and generates synthesized voice data to follow the fixed phrases audio data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device for controlling a conversation with a user. [Background technology]

[0002] In recent years, conversational robots that can have natural conversations with users have been attracting attention.

[0003] Patent Document 1 describes a control device for a conversation robot that can naturally hold a personal conversation depending on the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-168015 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the control device described in Patent Document 1 synthesizes text data, which is generated by a conversation engine and constitutes a conversation with a user, into voice data corresponding to the text data. Therefore, when synthesizing voice data, there is a long waiting time between inputting the text data and having the conversation robot speak the contents of the text data. This makes it difficult to have a responsive conversation with the user.

[0006] In view of the above problems, an object of the present invention is to realize natural conversation with a user. [Means for solving the problem]

[0007] The present invention employs the following technical solutions to solve the above problems. The reference symbols in parentheses in the claims and this section are merely examples showing the correspondence with the specific solutions described in the embodiments below as one aspect, and do not limit the technical scope of the present invention.

[0008] A control device (11) according to one aspect of the present invention is a control device for controlling a conversation with a user, and includes an input unit (114) that receives a voice input related to an utterance from the user, an output unit (117) that outputs a voice related to a response to the utterance to the user, a voice data storage unit (112a) that pre-stores voice data of a predetermined number of standard phrases used as openings of a response, a processing unit (111) that transmits a prompt including data of the utterance from the user to a natural language processing system (50) that uses a large-scale language model and receives a response from the natural language processing system, and and a speech synthesis unit (116) that synthesizes speech based on the answer from the utterance, wherein the processing unit includes a classifier (111a) that classifies which of the standard opening phrases of the response should be returned in response to the utterance and predicts the standard opening phrase of the response to the utterance according to the classification result, reads out speech data corresponding to the standard opening phrase of the response predicted by the classifier from the speech data storage unit, and causes the output unit to output speech based on the read out speech data, and the speech synthesis unit synthesizes speech related to the answer received from the natural language processing system and generates synthesized speech data following the speech data of the standard phrase.

[0009] With the above configuration, the control device can start a response to a user utterance by classifying the opening phrase of the response using a classifier, without querying the natural language processing system for the content of the response. Furthermore, if voice data corresponding to the opening phrase of the response predicted by the classifier is stored in the voice data storage unit, the voice data can be reproduced without performing voice synthesis using the voice synthesis unit. As a result, the control device according to this embodiment can speed up the opening of the response to a user utterance without waiting for the voice synthesis processing time of the voice synthesis unit. As a result, natural conversation with the user can be realized.

[0010] A robot system (100) according to one aspect of the present invention includes the control device (11) and a natural language processing system (50).

[0011] a processing step of transmitting a prompt including data of the utterance from the user to a natural language processing system utilizing a large-scale language model and receiving a response from the natural language processing system; a speech synthesis step of synthesizing a speech related to the response received from the natural language processing system and generating synthesized speech data following the speech data of the speech phrase; and a second output step of outputting a speech based on the speech data synthesized in the speech synthesis step.

[0012] a processing step of transmitting a prompt including data of the utterance from the user to a natural language processing system using a large-scale language model and receiving a response from the natural language processing system; a speech synthesis step of synthesizing a speech related to the response received from the natural language processing system and generating synthesized speech data following the speech data of the speech phrase; and a second output step of outputting a speech based on the speech data synthesized in the speech synthesis step. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide a control device and the like that can realize natural conversation with a user. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a robot system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the voice data stored in the voice data storage unit. [Figure 3A] FIG. 3A is a flowchart illustrating the flow of operations of the processing unit according to the first embodiment. [Figure 3B] FIG. 3B is a flowchart illustrating the flow of operations of the processing unit according to the first embodiment. [Figure 3C]FIG. 3C is a flowchart illustrating the flow of operations of the processing unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a configuration of a robot system according to the second embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a function list according to the second embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a format of a history log according to the second embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a history log according to the second embodiment. [Figure 8] FIG. 8 illustrates an example of preconditions according to the second embodiment. [Figure 9] FIG. 9 is a flowchart illustrating the flow of operations of the processing unit according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing the processing of the robot control unit according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below shows an example of how the present invention can be implemented, and the present invention is not limited to the specific configuration described below. When implementing the present invention, a specific configuration corresponding to the embodiment may be appropriately adopted.

[0016] (First embodiment) Fig. 1 is a diagram showing an example of the configuration of a robot system 100 according to a first embodiment. The robot system 100 shown in Fig. 1 includes a conversational robot 10 and a natural language processing system 50. The conversational robot 10 and the natural language processing system 50 are connected via a network so as to be able to send and receive data.

[0017] The conversational robot 10 includes a control device 11, a microphone 13, and a speaker 14.

[0018] The natural language processing system 50 is a natural language processing system that uses a large-scale language model (LLM). A large-scale language model is a deep learning model that is generated by pre-training based on a huge amount of data on what is called a language model that models natural language, spoken by humans, based on its occurrence probability. As the natural language processing system 50 that uses a large-scale language model, for example, GPT-3, GPT-3.5, or GPT-4 can be used. When the natural language processing system 50 receives a request, it uses the large-scale language model to statistically estimate the probability of generating the next word from the sentence included in the received request, and transmits the estimation result to the request source.

[0019] The robot system 100 of this embodiment is configured such that the conversational robot 10 transmits a prompt including data of an utterance from a user to the natural language processing system 50, synthesizes a voice corresponding to a response from the natural language processing system 50, and causes the conversational robot 10 to speak the voice.

[0020] The control device 11 is a device for controlling a conversation with a user, and includes a processing unit 111, a storage unit 112, a communication unit 113, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are connected to each other so that they can communicate with each other, for example, via a bus. In this embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may be part of the functional configuration of the processing unit 111.

[0021] The processing unit 111 is configured with, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) that is used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 112 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0022] The processing unit 111 transmits a prompt including data of an utterance from a user to, for example, a natural language processing system 50 that uses a large-scale language model, and receives a response from the natural language processing system 50. The processing unit 111 includes, as functional components, a classifier 111a and a search unit 111b.

[0023] The classifier 111a classifies a predetermined number of standard phrases into which opening phrases of a response should be returned in response to a user utterance, and predicts the opening phrase of a response to the utterance based on the classification result. The classifier 111a may be, for example, a neural network model. Specifically, the classification of text data by the classifier 111a corresponds to the output of a class into which the text data should be classified when the text data is input to the classifier 111a. The classifier 111a is trained using training data. More specifically, the classifier 111a is trained using a combination of training text data and labels (trainer classes) assigned to the training text data, which constitute the training data. The training data can be generated, for example, by using GPT to generate sample inputs corresponding to user utterances, and then using GPT to generate responses to the samples, and using the openings of the generated responses as training classes.

[0024] The search unit 111b searches for voice data corresponding to the standard phrase at the beginning of the response predicted by the classifier 111a from a voice data storage unit 112a (described later) in the storage unit 112. As will be described later, the voice data storage unit 112a stores voice data of a predetermined number of standard phrases in association with hash values ​​generated from the standard phrases. The search unit 111b uses the hash values ​​to search for voice data corresponding to the standard phrase at the beginning of the response to an utterance from the user from the voice data storage unit 112a. Details of the operations of the classifier 111a and the search unit 111b will be described later.

[0025] The storage unit 112 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as a hard disk drive (HDD) for storing various information and a solid state drive (SSD). Note that the storage unit 112 may also be a drive or the like that reads and writes various information from and to portable storage media such as a CD-ROM, a DVD, and a flash memory.

[0026] The storage unit 112 includes a voice data storage unit 112a and an input buffer 112b. The voice data storage unit 112a and the input buffer 112b may be cache memories provided within the processing unit 111. The voice data storage unit 112a will be described later. The input buffer 112b stores data of utterances from the user input from a voice input unit 114, which will be described later. The processing unit 111 includes the data read from the input buffer 112b in a prompt and transmits the prompt to the natural language processing system 50.

[0027] The communication unit 113 is realized by, for example, a circuit that connects to a network, and communicates with the natural language processing system 50 via the network.

[0028] The voice input unit 114 is connected by wire or wirelessly to the microphone 13 that collects voice and outputs a voice signal, and receives a voice signal from the microphone 13. In other words, the voice input unit 114 receives a voice input related to a speech from the user.

[0029] The speech recognition unit 115 performs speech recognition processing on the speech signal from the speech input unit 114 and outputs text information represented by the speech to the processing unit 111. The speech recognition unit 115 can use a speech recognition technology that converts speech signals into text information using deep learning technology. As an example, the speech recognition unit 115 can use the Azure (registered trademark) AI Speech service described below.

[0030] The speech synthesis unit 116 has a function of synthesizing speech data corresponding to an input sentence. The speech synthesis unit 116 performs speech synthesis of a response or a standard phrase from the natural language processing system 50. Specifically, the speech synthesis unit 116 synthesizes speech related to the response received from the natural language processing system 50, and generates synthesized speech data following speech data of the standard phrase at the beginning of the response. As an example, the Azure AI Speech service can be used for the speech synthesis unit 116.

[0031] The Azure AI Speech service is a cloud-based AI service that can convert speech to text (speech recognition) and text to speech (text-to-speech). Using this service, functions such as speech recognition and speech synthesis can be incorporated into applications and software. By using the Azure AI Speech service in the speech synthesis unit 116, the speech synthesis unit 116 synthesizes speech related to a response received from the natural language processing system 50 and generates speech data in WAV (Waveform Audio File Format) format.

[0032] The audio output unit 117 is connected by wire or wirelessly to the speaker 14 that outputs audio based on audio data to the outside, and reproduces audio based on the audio data from the processing unit 111 or the audio synthesis unit 116 and outputs it to the speaker 14. In other words, the audio output unit 117 outputs audio related to a response to an utterance from the user to the user.

[0033] The following describes the audio data storage unit 112a, which is a storage area provided in the storage unit 112.

[0034] (Audio data storage unit) The storage unit 112 includes an audio data storage unit 112a capable of storing audio data.

[0035] Currently, when synthesizing speech, there is a long waiting time from when text is input until the content of that text is spoken. This is one of the reasons why a conversation with a user cannot be responsive. Therefore, in this embodiment, the speech data storage unit 112a stores in advance a predetermined number of standard speech data used as openings of responses to speech from the user.

[0036] 2 shows an example of voice data of a predetermined number of standard phrases (voice data No. 1 to No. 23) used as the opening of a response, which are stored in advance in the voice data storage unit 112a, and voice data (voice data No. 24 and subsequent voice data) generated by the voice synthesis unit 116 and stored in the voice data storage unit 112a. The voice data No. 1 to No. 23 are voice data of a total of 23 standard phrases, namely "Hello," "Thank you," "Understood," "Sorry to keep you waiting," ..., and "Goodbye," and are stored in advance in the voice data storage unit 112a. In this embodiment, as an example, voice data of 23 standard phrases is stored in the voice data storage unit 112a, but this number is not limited to 23.

[0037] The predetermined number of template phrases are obtained by, for example, having GPT generate sample inputs corresponding to user utterances, having GPT generate responses to those samples, and then selecting a predetermined number of the beginnings of the generated responses in descending order of frequency of appearance. The voice data storage unit 112a stores the voice data of the predetermined number of template phrases in association with hash values ​​calculated using a hash function such as SHA-256.

[0038] The classifier 111a classifies a predetermined number of standard phrases into the standard opening phrases to be returned in response to a user utterance, and predicts the standard opening phrase of the response to the utterance based on the classification result. By classifying the standard opening phrases of the response to the user utterance using the classifier 111a, a response can be started without querying the natural language processing system 50 about the content of the response. If voice data corresponding to the standard opening phrase of the response predicted by the classifier 111a is stored in the voice data storage unit 112a, the processing unit 111 can read and play the voice data from the voice data storage unit 112a without performing voice synthesis using the voice synthesis unit 116. As a result, in this embodiment, the start of the response to the user utterance can be generated at high speed without waiting for the voice synthesis processing time of the voice synthesis unit 116. If the classifier 111a cannot classify the user's utterance into any of a predetermined number of standard phrases used as opening phrases of a response, it classifies the utterance into "other" and does not predict a standard phrase for opening a response.

[0039] 2 are examples of the voice data generated by the voice synthesis unit 116 and stored in the voice data storage unit 112a. In the voice data storage unit 112a, phrases generated by the voice synthesis unit 116, such as "It's a nice day today," "Thank you for coming today," and "What can I do for you today?", are stored and stored in association with their respective hash values.

[0040] 3A to 3C, a specific operation of processing unit 111 will be described.

[0041] In step S301, the processing unit 111 determines whether there is an input from the voice recognition unit 115, that is, whether there is an utterance from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S302. If there is no input from the voice recognition unit 115, the flow returns to step S301 and waits for an input from the voice recognition unit 115.

[0042] In step S302, the processing unit 111 overwrites the input from the speech recognition unit 115 in the input buffer 112b.

[0043] In step S303, the classifier 111a in the processing unit 111 classifies the data read from the input buffer 112b (i.e., data of an utterance from the user) to determine which of the standard opening phrases should be returned as a response, and predicts the standard opening phrase of the response to the utterance based on the classification result. If the classifier 111a can classify the data to determine which of the standard opening phrases should be returned as a response (i.e., if the utterance from the user is classified as one of a predetermined number of standard phrases other than "Other"), the processing unit 111 performs the processes from step S304 onwards and step S309 onwards in parallel. If the classifier 111a cannot classify the data to determine which of the standard opening phrases should be returned as a response (i.e., if the utterance from the user is classified as "Other"), the processing unit 111 only performs the processes from step S309 onwards.

[0044] In step S304, the search unit 111b in the processing unit 111 searches the voice data storage unit 112a for voice data corresponding to the standard opening phrase of the predicted response. If the search by the search unit 111b finds voice data corresponding to the standard opening phrase of the predicted response in the voice data storage unit 112a, the flow proceeds to step S305. If the search by the search unit 111b does not find voice data corresponding to the standard opening phrase of the predicted response in the voice data storage unit 112a, the flow proceeds to step S306.

[0045] In step S305, if the search unit 111b finds speech data corresponding to the opening template of the predicted response in the speech data storage unit 112a through a search, the processing unit 111 plays the speech data and outputs the speech from the speech output unit 117. If there is no sentence following the opening template of the response, the flow returns to step S301 and waits for the next input from the speech recognition unit 115.

[0046] In step S306, if the search unit 111b does not find any voice data corresponding to the opening template of the predicted response in the voice data storage unit 112a, the processing unit 111 causes the voice synthesis unit 116 to generate voice data corresponding to the opening template of the predicted response.

[0047] In step S307, the processing unit 111 plays back the voice data corresponding to the template sentence at the beginning of the predicted response, which was generated in step S306, and causes the voice output unit 117 to output the voice.

[0048] In step S308, the processing unit 111 stores and accumulates the voice data corresponding to the predicted opening template of the response generated in step S306 in the voice data storage unit 112a in association with the hash value. If there is no sentence following the opening template of the response, the flow returns to step S301 and waits for the next input from the voice recognition unit 115.

[0049] The processes of steps S306 to S308 are performed when the search unit 111b fails to find voice data of a predetermined number of standard phrases (standard phrases No. 1 to No. 23 in the example of FIG. 2) that are used as openings of a response and that are predicted by classifying utterances from a user and predicted by the classifier 111a. Typically, voice data corresponding to standard phrases is stored in the voice data storage unit 112a. For example, when changing the voice setting of the conversational robot 10 from a male voice to a female voice or when changing the classifier 111a, the voice data in the voice data storage unit 112a may be cleared, and therefore the processes of steps S306 to S308 may be necessary. When the processes of steps S306 to S308 are performed, it is necessary to synthesize the opening voice of a response. Therefore, it is generally preferable to prepare voice data of a predetermined number of standard phrases in advance.

[0050] In step S309, the processing unit 111 transmits a prompt including the data read from the input buffer 112b (i.e., data of the utterance from the user) to the natural language processing system 50. At this time, if the processing unit 111 classifies the utterance from the user into a category other than "other" in step S303, the processing unit 111 includes in the prompt an instruction to create a continuation of the standard opening phrase of the response predicted in step S303. For example, if the standard opening phrase of the response predicted by the classifier 111a is "Hello," the processing unit 111 transmits a prompt to the natural language processing system 50 to generate a continuation of "Hello" as a response. On the other hand, if the processing unit 111 classifies the utterance from the user into "other" in step S303, the processing unit 111 was unable to predict a standard opening phrase of the response in step S303, and therefore transmits a prompt to the natural language processing system 50 to generate an entire sentence. Furthermore, in step S309, if the conversation between the user and the conversational robot 10 is continuing and the user makes an utterance in response to an utterance from the conversational robot 10, the last utterance from the conversational robot 10 may be input to the input buffer 112b in addition to the user's utterance. This allows a response that maintains the continuity of the conversation to be received from the natural language processing system 50.

[0051] In step S310, the processing unit 111 receives a response from the natural language processing system 50.

[0052] In step S311, the processing unit 111 causes the search unit 111b to search for corresponding speech data from the speech data storage unit 112a for each unit separated by punctuation marks (hereinafter referred to as a "punctuation mark unit") in the response received from the natural language processing system 50. For example, if the response from the natural language processing system 50 is "Thank you for coming today. What can we do for you today?", the processing unit 111 first sets "Thank you for coming today" as a punctuation mark unit and causes the search unit 111b to search for speech data corresponding to the punctuation mark unit from the speech data storage unit 112a. Note that the punctuation marks referred to here include not only the period "." and the comma "," but also the question mark "?" and the exclamation mark "!".

[0053] If the search unit 111b finds speech data corresponding to the punctuation unit in the speech data storage unit 112a through a search, the flow proceeds to step S312. If the search unit 111b does not find speech data corresponding to the punctuation unit in the speech data storage unit 112a through a search, the flow proceeds to step S314.

[0054] In step S312, if the search unit 111b searches and finds audio data corresponding to the punctuation unit in the audio data storage unit 112a, the processing unit 111 plays the audio data and causes the audio output unit 117 to output the audio.

[0055] In step S313, the processing unit 111 checks whether or not one of the two flow processes branching off depending on the search results in step S311 has been executed for all punctuation units. If one of the two flow processes has not been completed for all punctuation units, the flow returns to step S311. If one of the two flow processes branching off for all punctuation units has been completed in step S313, the flow returns to step S301 and waits for the next input from the speech recognition unit 115. In the current example, the processing unit 111 has not completed one of the two flow processes branching off for the punctuation unit "What can I do for you today?", so the flow returns to step S311.

[0056] In step S314, if the search unit 111b does not find any audio data corresponding to the punctuation unit in the audio data storage unit 112a, the processing unit 111 causes the audio synthesis unit 116 to generate audio data corresponding to the punctuation unit.

[0057] In step S315, the processing unit 111 plays back the audio data corresponding to the punctuation unit generated in step S314, and causes the audio output unit 117 to output the audio.

[0058] In step S316, the processing unit 111 stores and accumulates the voice data corresponding to the punctuation unit generated in step S314 in the voice data storage unit 112a in association with the hash value.

[0059] Step S317 is similar to the processing in step S313, and therefore a detailed description thereof will be omitted.

[0060] Through the above processing, the control device 11 according to the present embodiment can start a response without querying the natural language processing system 50 for the content of the response by classifying the standard opening phrase of a response to an utterance from a user using the classifier 111a. Furthermore, if voice data corresponding to the standard opening phrase of a response predicted by the classifier 111a is stored in the voice data storage unit 112a, the voice data can be reproduced without performing voice synthesis using the voice synthesis unit 116. This allows the control device 11 according to the present embodiment to speed up the beginning of a response to an utterance from a user without waiting for the voice synthesis processing time of the voice synthesis unit 116. Furthermore, while reproducing the voice of the beginning of the response, the processing unit 111 causes the voice synthesis unit 116 to synthesize the subsequent sentences following the beginning of the response, separated by punctuation marks. This speeds up the overall speech of the conversational robot 10 and enables the conversational robot 10 to speak in a seamless, natural flow. Furthermore, by making the conversational robot 10 speak in punctuation units, the intonation of the speech can be maintained. As a result, the control device 11 according to this embodiment can realize a natural conversation with the user.

[0061] As described above, in the control device 11 according to the present embodiment, the voice data storage unit 112a stores and accumulates the voice data generated by the voice synthesis unit 116. As a result, the voice data generated by the voice synthesis unit 116 is accumulated in the voice data storage unit 112a, so that the search by the search unit 111b in step S309 of Fig. 3C makes it easier to find voice data corresponding to a punctuation mark unit in the voice data storage unit 112a. When voice data corresponding to a punctuation mark unit is found in the voice data storage unit 112a, it is no longer necessary to generate the voice data by voice synthesis. This eliminates the need to wait for the voice synthesis processing time, and the overall speech by the conversational robot 10 can be made faster.

[0062] Furthermore, in the control device 11 according to this embodiment, when the search unit 111b finds voice data corresponding to a standard phrase at the beginning of a response to a user's utterance in the voice data storage unit 112a through a search (i.e., when the classifier 111a can classify the user's utterance into one of the predetermined standard phrases), the processing unit 111 may insert a predetermined length of pause (e.g., 0.5 seconds, 1 second, 1.5 seconds, etc.) before causing the voice output unit 117 to output the voice corresponding to the standard phrase at the beginning of the response. This prevents the voice output unit 117 from instantly playing back the voice corresponding to the voice data when the voice data corresponding to the standard phrase at the beginning of the response is found in the voice data storage unit 112a, thereby realizing a natural conversation with an appropriate pause. Specifically, in response to the user's utterance "Hello," the conversational robot 10 may return the response "Hello" after an appropriate pause, rather than immediately.

[0063] Furthermore, in the control device 11 according to this embodiment, the processing unit 111 may transmit to the natural language processing system 50 a prompt containing an instruction to cause the natural language processing system 50 to return an answer consisting of a predetermined number of characters or less. The predetermined number of characters may be set to 10, 15, 20, 25, 30, 35, 40, or other characters depending on the situation. This allows the answer generated by the natural language processing system to be a collection of short sentences consisting of a predetermined number of characters or less. The voice data storage unit 112a stores voice data of these short sentences consisting of a predetermined number of characters or less. As a result, the search unit 111b is more likely to find a hit for voice data corresponding to a punctuation mark unit, thereby further speeding up the overall speech of the conversational robot 10.

[0064] (Second embodiment) Next, a robot system 200 according to a second embodiment will be described. In the second embodiment, a robot system 200 will be described in which an arm robot is combined with the conversation robot 10 according to the first embodiment.

[0065] FIG. 4 is a diagram showing an example of the configuration of a robot system 200 according to this embodiment. The same components as those in the robot system 100 according to the first embodiment are given the same reference numbers, and their description may be omitted. The robot system 200 shown in FIG. 4 includes a working robot 20, a natural language processing system 50, and a search server 60. The working robot 20 and the natural language processing system 50 are connected so as to be able to send and receive data via a network. The working robot 20 and the search server 60 are connected so as to be able to send and receive data via the network.

[0066] The working robot 20 includes a control device 21, an arm robot 12, a microphone 13, and a speaker 14.

[0067] The natural language processing system 50 is a natural language processing system that uses a large-scale language model (LLM). A large-scale language model is a deep learning model that is generated by pre-training based on a huge amount of data on what is called a language model that models natural language, spoken by humans, based on its occurrence probability. As the natural language processing system 50 that uses a large-scale language model, for example, GPT-3, GPT-3.5, or GPT-4 can be used. When the natural language processing system 50 receives a request, it uses the large-scale language model to statistically estimate the probability of generating the next word from the sentence included in the received request, and transmits the estimation result to the request source.

[0068] The search server 60 performs a search process based on a search request (an example of an operation command) sent from the work robot 20, and provides information corresponding to the search request by sending a search result (an example of an execution result) indicating the result of the search process to the work robot 20.

[0069] In the robot system 200 of this embodiment, the work robot 20 sends a prompt including data of an utterance from a user to the natural language processing system 50, synthesizes a voice relating to the response from the natural language processing system 50, and causes the work robot 20 to speak. At the same time, a prompt inquiring about the operation of the arm robot 12 in the work robot 20 is sent, and the arm robot 12 is operated based on the response from the natural language processing system 50.

[0070] The control device 21 is a control device for controlling the working robot 20 based on interaction with a user, and includes a processing unit 111, a memory unit 212, a communication unit 113, a voice input unit 114, a voice recognition unit 115, a voice synthesis unit 116, and a voice output unit 117. These are connected to each other so that they can communicate with each other, for example, via a bus. In this embodiment, the processing unit 111, the voice recognition unit 115, and the voice synthesis unit 116 are each configured separately, but the voice recognition unit 115 and the voice synthesis unit 116 may also be part of the functional configuration of the processing unit 111.

[0071] The processing unit 111 is configured by, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory) that is used as a main storage device when the CPU executes processing. The CPU, for example, loads a program stored in the storage unit 212 into the RAM and executes the program, thereby realizing various functions corresponding to the program.

[0072] The processing unit 111, for example, sends prompts to a natural language processing system 50 that uses a large-scale language model, receives a response from the natural language processing system 50, and executes a functional module (described later) identified based on the received response, causing the working robot 20 to speak, the arm robot 12 to operate, and the search server 60 to perform a search. The processing unit 111 includes, as functional components, a classifier 111a and a search unit 111b.

[0073] The classifier 111a classifies a predetermined number of standard phrases into which opening phrases of a response should be returned in response to a user utterance, and predicts the opening phrase of a response to the utterance based on the classification result. The classifier 111a may be, for example, a neural network model. Specifically, the classification of text data by the classifier 111a corresponds to the output of a class into which the text data should be classified when the text data is input to the classifier 111a. The classifier 111a is trained using training data. More specifically, the classifier 111a is trained using a combination of training text data and labels (trainer classes) assigned to the training text data, which constitute the training data. The training data can be generated, for example, by using GPT to generate sample inputs corresponding to user utterances, and then using GPT to generate responses to the samples, and using the openings of the generated responses as training classes.

[0074] The search unit 111b searches for voice data corresponding to the standard phrase at the beginning of the response predicted by the classifier 111a from a voice data storage unit 112a (described later) in the storage unit 112. As will be described later, the voice data storage unit 112a stores voice data of a predetermined number of standard phrases in association with hash values. The search unit 111b uses the hash values ​​to search for voice data corresponding to the standard phrase at the beginning of the response to an utterance from the user from the voice data storage unit 112a. Details of the operations of the classifier 111a and the search unit 111b are the same as those in the first embodiment, and therefore will not be described again.

[0075] The storage unit 212 is a so-called auxiliary storage device that includes a nonvolatile storage circuit such as a hard disk drive (HDD) for storing various information and a solid state drive (SSD). Note that the storage unit 212 may also be a drive or the like that reads and writes various information from and to portable storage media such as a CD-ROM, a DVD, and a flash memory.

[0076] Similar to storage unit 112 according to the first embodiment, storage unit 212 includes voice data storage unit 112a and input buffer 112b. Storage unit 212 according to the present embodiment further includes function list storage unit 112c, history log storage unit 112d, and prerequisite storage unit 112e. Function list storage unit 112c, history log storage unit 112d, and prerequisite storage unit 112e will be described later.

[0077] The communication unit 113 is realized by, for example, a circuit that connects to a network, and communicates with the natural language processing system 50 via the network.

[0078] The voice input unit 114 is connected by wire or wirelessly to the microphone 13 that collects voice and outputs a voice signal, and receives a voice signal from the microphone 13. That is, the voice input unit 114 receives voice input as a speech-related instruction from the user.

[0079] The speech recognition unit 115 performs speech recognition processing on the speech signal from the speech input unit 114 and outputs text information represented by the speech to the processing unit 111. The speech recognition unit 115 can use a speech recognition technology that converts speech signals into text information using deep learning technology. As an example, the speech recognition unit 115 can use the Azure AI Speech service described above.

[0080] The speech synthesis unit 116 has a function of synthesizing speech data corresponding to an input sentence. The speech synthesis unit 116 performs speech synthesis of a response or a standard phrase from the natural language processing system 50. The speech synthesis unit 116 performs speech synthesis by executing a functional module identified by the processing unit 111 based on the response from the natural language processing system 50. Specifically, the speech synthesis unit 116 synthesizes speech related to the response received from the natural language processing system 50 by executing the functional module, and generates synthesized speech data following speech data of the standard phrase at the beginning of the response. As an example, the speech synthesis unit 116 can use the Azure AI Speech service described above.

[0081] The audio output unit 117 is connected by wire or wirelessly to the speaker 14 that outputs audio based on audio data to the outside, and reproduces audio based on the audio data from the processing unit 111 or the audio synthesis unit 116 and outputs it to the speaker 14. In other words, the audio output unit 117 outputs audio related to a response to an utterance from the user to the user.

[0082] The arm robot 12 includes a robot control unit 121 and a plurality of axes including a first axis 121a, a second axis 121b, a third axis 121c, and a fourth axis 121d.

[0083] The robot control unit 121 receives operation commands for the arm robot 12 generated by the processing unit 111 executing the functional modules, and controls each part of the arm robot 12 according to the operation commands. Specifically, the robot control unit 121 is a computer equipped with hardware such as a calculation unit such as a CPU, a main storage unit such as a semiconductor memory, an auxiliary storage unit such as a hard disk, and a communication unit.

[0084] The multiple axes including the first axis 121a, the second axis 121b, the third axis 121c, and the fourth axis 121d are rotation axes for moving arm joints (not shown) provided on the arm robot 12 using motors (not shown). By independently controlling these multiple axes, the arm robot 12 can be made to perform an intended operation.

[0085] The following describes the function list storage unit 112c, the history log storage unit 112d, and the prerequisite storage unit 112e, which are storage areas provided in the storage unit 212.

[0086] (Feature List) The memory unit 212 includes a function list memory unit 112c that stores a function list. Here, a functional module is a packaged series of general-purpose operations of the working robot 20. The function list is a compilation of usage methods for each functional module, which is provided to the natural language processing system 50 as prior information when the natural language processing system 50 is caused to generate functional modules and parameters suitable for task processing. In other words, the function list is a list containing multiple functional modules, each of which defines as a functional module a number of predetermined operations to be performed by the working robot 20.

[0087] Currently, it is difficult to generate a correct program (e.g., low code) for the work robot 20 from scratch by sending a request in natural language to the natural language processing system 50. For this reason, simply sending a request in natural language to the natural language processing system 50 will not allow the work robot 20 to perform the intended action.

[0088] Therefore, in this embodiment, the processing unit 111 transmits to the natural language processing system 50 a prompt containing the function list stored in the function list storage unit 112c in addition to the instruction in natural language from the user.

[0089] This allows the natural language processing system 50 to determine (or identify) from the function list the functional module that should be executed by the work robot 20 based on the user's instructions. As described above, the functional module defines the predetermined operation that the work robot 20 is to perform, so the processing unit 111 can identify the functional module based on the response from the natural language processing system 50 regarding which functional module from the function list should be executed, and can cause the work robot 20 to execute the identified functional module. This allows the work robot 20 to appropriately perform the intended operation.

[0090] The functional modules also receive as input parameters related to the coordinates and / or time required for the operation of the work robot 20, or parameters related to the text to be output to the user. In this case, the processing unit 111 sends the parameters to be used by each functional module, including them in a function list, to the natural language processing system 50. The processing unit 111 then receives parameter data to be given to the functional module along with data identifying the functional module from the natural language processing system 50, and gives the parameter data to be given to the identified functional module. Even in this case, the data identifying the functional module and the parameter data to be given to the functional module are sent as responses from the natural language processing system 50, so the operations of the work robot 20 are limited to those specified in the functional module, and the robot can be operated more appropriately using parameter data that matches the functional module.

[0091] 5 is a diagram showing an example of a function list. The function module Speak is a function module that converts a specified sentence (sentence = an example of a parameter) into voice and outputs it to the speaker 14. The function module PickAndGive is a function module that causes the arm robot 12 to pick up a specified item (item = an example of a parameter) and perform an action of handing the picked item to the user. The function module Weather is a function module that causes the search server 60 to search for the current weather using, for example, the search engine Bing (registered trademark) and obtain the search results.

[0092] In the function list shown in Figure 5, the function name keys and JSON values ​​are written in a simple format because they are complicated, but the function module can be implemented in JSON format.

[0093] (History log) The storage unit 212 further includes a history log storage unit 112d that accumulates a history log.

[0094] The input buffer 112b stores data of instructions input by the user via the voice input unit 114. If data is already stored in the input buffer 112b, it is overwritten. Furthermore, the input buffer 112b is overwritten with the execution result when the function module identified by the response from the natural language processing system 50 is executed.

[0095] The history log is log data including data read from the input buffer 112b, an output from the classifier 111a obtained in response to the read data, and a response from the natural language processing system 50 obtained in response to the read data. In this way, a set consisting of an input to the classifier 111a and the natural language processing system 50 and an output in response to the input is added to the history log storage unit 112d as a history log.

[0096] In this embodiment, after transmitting an instruction in natural language from the user, the processing unit 111 includes the data read from the input buffer 112b and the history log accumulated in the history log storage unit 112d in a prompt and transmits the prompt to the natural language processing system 50.

[0097] As a result, the natural language processing system 50 sends a response based on the data read from the input buffer 112b and the history log, and the processing unit 111 can cause the work robot 20 to perform continuous operations based on past interactions based on the history log.

[0098] FIG. 6 shows an example of the history log format. In FIG. 6, the history log format has four fields: "speaker," "predicted template," "utterance content," and "function call." The "speaker" refers to a user who speaks to the work robot 20, or the processing unit 111 that executes the function module Speak to cause the work robot 20 to speak. The "predicted template" refers to a template that starts a response predicted by the classifier 111a for data read from the input buffer 112b. The "utterance content" refers to the content of the user's utterance when the "speaker" is a user. When the "speaker" is the processing unit 111, the "utterance content" refers to the content following the predicted template that the work robot 20 is caused to utter by executing the function module Speak identified by the natural language processing system 50 for the data read from the input buffer 112b. The "function call" refers to a function module other than Speak identified by the natural language processing system 50 for the data read from the input buffer 112b.

[0099] FIG. 7 is a diagram showing an example of a history log. The first line is log data indicating that the user uttered "Hello" to the work robot 20 (specifically, the microphone 13). If the "speaker" is the user, the "predicted template" and "function call" fields are left blank. The second line is log data indicating that the classifier 111a (i.e., the processing unit 111) predicted a template for the opening of a response to the user's utterance of "Hello," and that the natural language processing system 50 selected the function module Speak from the function list in response to the user's utterance of "Hello," and determined "What can I do for you today?" following the predicted template "Hello" as parameter data to be given to the function module Speak. In this way, the log data on the first and second lines are input to the classifier 111a and the natural language processing system 50, respectively, and the output corresponding to the input, and are added as a set to the history log storage unit 112d.

[0100] The third line is log data indicating that the user uttered "Please give me some tea" to the work robot 20. The fourth line is log data indicating that the classifier 111a predicted the opening template phrase of the response "I understand" in response to the user's utterance "Please give me some tea," and that the natural language processing system 50 determined the functional module Speak from the function list in response to the utterance "Please give me some tea," and determined "Please wait a moment" following the predicted template phrase "I understand" as parameter data to be given to the functional module Speak, as well as determined the functional module PickAndGive, and determined "tea" as the parameter to be given to the functional module PickAndGive.

[0101] The fifth line is log data showing that the classifier 111a used the execution result of the functional module PickAndGive on the fourth line as input to predict the opening phrase of the response, "Sorry to keep you waiting," and the natural language processing system 50 selected the functional module Speak from the function list, determined the predicted phrase "Sorry to keep you waiting" to be followed by "Here's some tea. Please have some," as parameter data to be given to the functional module Speak, and called the functional module Speak for each punctuation unit, "Here's some tea." and "Please have some."

[0102] The sixth line is log data indicating that the user uttered "Thank you. That's wonderful" to the work robot 20. The seventh line is log data indicating that the classifier 111a, using the content of the sixth line as input, predicted the standard phrase that would begin the response "Thank you," and that the natural language processing system 50 selected the function module Speak from the function list, determined "Please let me know if there is anything I can do to help," which follows the predicted standard phrase "Thank you," as parameter data to be given to the function module Speak, and called the function module Speak for each punctuation mark.

[0103] Line 8 is log data indicating that the user uttered "Please check the weather" to the work robot 20. Line 9 is log data indicating that the classifier 111a, using the content of line 8 as input, predicted the opening template of the response, "I understand," and the natural language processing system 50 determined the functional module Weather. In line 9, the natural language processing system 50 did not generate a sentence following the predicted template "I understand," so the "utterance content" field is blank.

[0104] Line 10 is log data showing that the natural language processing system 50, using the execution result of the functional module Weather on line 9 as input (i.e., the information "sunny" obtained as a search result), selected the functional module Speak from the function list, determined "It's sunny today. Looks like it's going to be hot" as parameter data to be given to the functional module Speak, and called the functional module Speak for each punctuation mark. On line 10, the classifier 111a, using the execution result of the functional module Weather on line 9 as input, was unable to predict the opening template of the response, so the "predicted template" is blank.

[0105] Line 11 is log data indicating that the user uttered "Tell me about yourself" to the work robot 20. Line 12 is log data indicating that, using the content of line 11 as input, the classifier 111a predicted the opening phrase of the response, "I understand," and the natural language processing system 50 selected the function module Speak from the function list, determined the parameter data to be given to the function module Speak as follows: "I'm an AI assistant. I can follow your instructions and perform tasks such as delivering items and performing web searches. What can I do for you? Please let me know.", and called the function module Speak for each punctuation mark.

[0106] (Prerequisite) The storage unit 212 further includes a precondition storage unit 112e that stores preconditions for the operation of the working robot 20.

[0107] The preconditions include the work procedures of the robot 20. Specifically, the work procedures of the working robot 20 define the operational guidelines for each stage of the operations to be performed by the working robot 20, which are divided into multiple stages. These operational guidelines may be written in a natural language such as Japanese or English, or in the Unified Modeling Language (UML).

[0108] When the work procedures of the work robot 20 are given to the natural language processing system 50 as preconditions, if the preconditions are written in source code, the natural language processing system 50 will follow the source code, resulting in a highly precise answer but not a flexible answer.

[0109] Therefore, in this embodiment, the processing unit 111 transmits to the natural language processing system 50 a prompt containing preconditions stored in the precondition storage unit 112e in addition to instructions in natural language from the user.

[0110] The preconditions divide the actions to be performed by the work robot 20 into multiple stages and specify the action plan for each stage, so the natural language processing system 50 can provide flexible yet appropriate answers in accordance with the action plans. Therefore, based on these answers, the processing unit 111 can cause the work robot 20 to perform appropriate actions in accordance with the action plans while ensuring flexibility in the work robot's actions.

[0111] FIG. 8 is a diagram showing an example of a precondition. In area 8a, "Your Role" is defined as a precondition. For example, area 8a contains instructions for the natural language processing system 50 to return an answer in which the number of characters in one sentence is less than a predetermined number, as described in the first embodiment. In area 8b, a "Workspace" is defined as a precondition. This is the workspace in which the working robot 20 operates, and various information about objects that the arm robot 12 can grasp is defined. In area 8c, a "Task" is defined as a precondition. This is the work procedure for the operation to be performed by the arm robot 12.

[0112] Next, a specific operation of the processing unit 111 will be described with reference to Fig. 9. Fig. 9 is a flowchart for explaining the flow of the operation of the processing unit 111.

[0113] In step S901, processing unit 111 reads the function list stored in function list storage unit 112c and the prerequisites stored in prerequisite storage unit 112e.

[0114] In step S902, the processing unit 111 determines whether there is an input from the voice recognition unit 115, that is, whether there is an instruction from the user via the microphone 13. If there is an input from the voice recognition unit 115, the flow proceeds to step S903. If there is no input from the voice recognition unit 115, the flow returns to step S902 and waits for an input from the voice recognition unit 115.

[0115] In step S903, processing unit 111 overwrites input buffer 112b with input from voice recognition unit 115. When input buffer 112b is overwritten in step S903, the processing from step S303 onwards in FIG. 3A is carried out with regard to processing of the speech of working robot 20, and voice data is reproduced by execution of functional module Speak by processing unit 111 in steps S305, S307, S312, and S315. In this embodiment, although not shown in FIGS. 3A to 3C, after the voice data has been reproduced, the execution result of functional module Speak is overwritten in input buffer 112b.

[0116] In step S904, the processing unit 111 transmits the function list and prerequisites read in step S901, as well as the data overwritten in the input buffer 112b and the history log accumulated in the history log storage unit 112d, to the natural language processing system 50. If no history log has been accumulated yet, a history log of empty data will be transmitted.

[0117] In step S905, the processing unit 111 adds the data in the input buffer 112b and the response from the natural language processing system 50 to the history log.

[0118] In step S906, the processing unit 111 branches the processing depending on the response from the natural language processing system 50. If the response from the natural language processing system 50 is to have the arm robot 12 perform an operation (i.e., execute the functional module PickAndGive), the flow proceeds to step S907. If the response from the natural language processing system 50 is to have the search server 60 perform a weather search (i.e., execute the functional module Weather), the flow also proceeds to step S909. If the response from the natural language processing system 50 is "completed," the flow returns to step S902 and waits for the next input from the voice recognition unit 115.

[0119] In step S907, the processing unit 111 executes the function module (here, the function module PickAndGive) identified based on the response from the natural language processing system 50, and transmits the generated operation command to the robot control unit 121.

[0120] In step S908, the processing unit 111 overwrites the input buffer 112b with the execution result obtained by executing the function module PickAndGive. Then, the flow returns to step S904, and the processing unit 111 transmits the function list, the prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50.

[0121] In step S909, the processing unit 111 executes the function module (here, the function module Weather) identified based on the response from the natural language processing system 50, and transmits the generated search request to the search server 60.

[0122] In step S910, the processing unit 111 overwrites the input buffer 112b with the search results obtained by executing the function module Weather. Then, the flow returns to step S904, and the processing unit 111 transmits the function list, the prerequisites, the data overwritten in the input buffer 112b, and the history log to the natural language processing system 50.

[0123] In addition, in Figure 9, an example has been described in which the processing unit 111 includes the data of the input buffer 112b, the function list, the history log, and the prerequisites in the prompt and transmits it to the natural language processing system 50. However, the processing unit 211 may also include at least one of these in addition to the data of the input buffer 112b in the prompt and transmit it to the natural language processing system 50.

[0124] For example, if the prompt includes all of the data in the input buffer 112b, the function list, the history log, and the prerequisites, the prerequisites that define the operating policy can be used to cause the work robot 20 to perform flexible and appropriate operations. The history log can be used to loop processing and cause the work robot 20 to perform continuous operations that take past events into consideration. Furthermore, the function list can be used to cause the work robot 20 to perform intended operations. Furthermore, the function list can be used to return an answer from the natural language processing system 50 as to which function module should be executed. Since data identifying a predetermined function module is obtained as the answer, it can be accumulated as a history log in a format that can be used later. Therefore, the history log can be included in the prompt, and the next answer can be obtained appropriately.

[0125] Next, a specific operation of the robot control unit 121 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the processing of the robot control unit 121.

[0126] In step S1001, the robot control unit 121 determines whether or not there is an input from the processing unit 111. If there is an input from the processing unit 111, that is, an operation command for the arm robot 12, the flow proceeds to step S1002. If there is no input from the function module execution unit 161a, the flow returns to step S1001 and waits for an input from the processing unit 111.

[0127] In step S1002, the robot control unit 121 executes the operation instructed in the operation command.

[0128] In step S1003, the robot control unit 121 notifies the processing unit 111 of the end of operation. Upon receiving this notification of the end of operation, the processing unit 111 overwrites the input buffer 112b with the execution result of the functional module in step S908. Note that the processing unit 111 may overwrite the input buffer 112b with the execution result of the functional module at the time of completing the execution of the functional module, without waiting for this notification of the end of operation.

[0129] (Other embodiments) The above describes embodiments of the present disclosure, but the present disclosure should not be construed as being limited to the above-described embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0130] Furthermore, the processing flow described in each of the above embodiments is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0131] It may be provided in the following manner. (Aspect 1) A control device (11) for controlling a conversation with a user, an input unit (114) that accepts voice input related to speech from the user; an output unit (117) that outputs a voice related to a response to the utterance to the user; a voice data storage unit (112a) that stores in advance voice data of a predetermined number of standard phrases used as openings of responses; a processing unit (111) that transmits a prompt including data of the utterance from the user to a natural language processing system (50) that utilizes a large-scale language model, and receives a response from the natural language processing system; a speech synthesis unit (116) that performs speech synthesis based on a response from the natural language processing system; and The processing unit a classifier (111a) that classifies which of the standard opening phrases of the response should be returned in response to the utterance, and predicts the standard opening phrase of the response to the utterance in accordance with the classification result; reads out from the voice data storage unit voice data corresponding to the opening template of the response predicted by the classifier, and outputs voice to the output unit based on the read out voice data; a control device that causes the speech synthesis unit to synthesize speech related to the response received from the natural language processing system and generate synthesized speech data following the speech data of the template.

[0132] (Aspect 2) The processing unit a search unit (111b) that searches the voice data storage unit for voice data corresponding to the predicted standard phrase at the beginning of the response; The processing unit The control device according to aspect 1, wherein when the search unit finds voice data corresponding to the opening template of the response in the voice data storage unit through a search, the control device causes the output unit to output the voice corresponding to the opening template of the response.

[0133] (Aspect 3) The processing unit The control device according to aspect 2, wherein if the search unit does not find voice data corresponding to the opening template of the response in the voice data storage unit through a search, the control device causes the voice synthesis unit to generate voice data corresponding to the opening template of the response.

[0134] (Aspect 4) the response received from the natural language processing system is delimited by punctuation; The processing unit causing the search unit to search within the voice data storage unit for each punctuation mark unit separated by a punctuation mark in the answer; When the search unit finds the voice data corresponding to the punctuation unit in the voice data storage unit through the search, the output unit outputs the voice corresponding to the punctuation unit; The control device according to aspect 2 or 3, wherein if the search unit does not find voice data corresponding to the punctuation unit in the voice data storage unit through a search, the control device causes the voice synthesis unit to generate voice data corresponding to the punctuation unit.

[0135] (Aspect 5) 5. The control device according to aspect 3 or 4, wherein the processing unit stores the voice data generated by the voice synthesis unit in the voice data storage unit.

[0136] (Aspect 6) The control device according to any one of aspects 1 to 5, wherein the processing unit includes in the prompt an instruction to create a follow-up sentence to the standard opening phrase of the response predicted by the classifier and transmits the prompt to the natural language processing system.

[0137] (Aspect 7) The control device according to aspect 2 or 3, wherein when the search unit finds voice data corresponding to the opening template of the response in the voice data storage unit through a search, the processing unit inserts a predetermined length of pause before outputting the voice corresponding to the opening template of the response to the output unit, and then outputs the voice corresponding to the opening template of the response to the output unit.

[0138] (Aspect 8) the voice data storage unit stores voice data of a standard phrase at the beginning of the response in association with a hash value, 3. The control device according to aspect 2, wherein the search unit searches the audio data storage unit for the audio data using the hash value.

[0139] (Aspect 9) the processing unit stores the voice data generated by the voice synthesis unit in association with a hash value; 3. The control device according to aspect 2, wherein the search unit searches the audio data storage unit for the audio data using the hash value.

[0140] (Aspect 10) The control device according to any one of aspects 1 to 9, wherein the processing unit includes in the prompt an instruction to the natural language processing system to return the answer consisting of a sentence with a predetermined number of characters or less, and transmits the instruction to the natural language processing system.

[0141] (Aspect 11) A control device according to any one of aspects 1 to 10; A robot system having the natural language processing system.

[0142] (Aspect 12) A control device for controlling a robot based on an interaction with a user, comprising: an input unit that receives instructions from the user to the robot; an output unit that outputs a voice related to a response to the instruction to the user; a voice data storage unit that stores in advance voice data of a predetermined number of standard phrases used as openings of responses; an input buffer for storing the instruction data input from the input unit; a processing unit that sends a prompt including the data read from the input buffer to a natural language processing system utilizing a large-scale language model and receives a response from the natural language processing system; a speech synthesis unit that performs speech synthesis based on a response from the natural language processing system; and The processing unit a classifier that classifies the data read from the input buffer to determine which of the standard opening phrases of the response should be returned, and predicts the standard opening phrase of the response to the instruction in accordance with the classification result; a control device that reads out from the voice data storage unit voice data corresponding to the opening template of the response predicted by the classifier, causes the output unit to output voice based on the read voice data, and causes the voice synthesis unit to synthesize voice related to the answer received from the natural language processing system, and generates synthesized voice data following the voice data of the template.

[0143] (Aspect 13) 1. A control method for controlling a conversation with a user, comprising: an input step of accepting a voice input related to a speech from the user; a classification step of classifying the utterance into which of the standard opening phrases should be returned, and predicting the standard opening phrase of the response to the utterance according to the classification result; a first output step of reading out voice data corresponding to the standard phrases at the beginning of the response predicted by the classification step from a voice data storage unit that stores voice data of a predetermined number of standard phrases used as the beginning of the response in advance, and outputting voice based on the read voice data; a processing step of sending a prompt including data of the utterance from the user to a natural language processing system utilizing a large-scale language model, and receiving a response from the natural language processing system; a speech synthesis step of synthesizing a speech related to the response received from the natural language processing system and generating the synthesized speech data following the speech data of the template; a second output step of outputting a voice based on the voice data synthesized in the voice synthesis step.

[0144] (Aspect 14) A control program for controlling a conversation with a user, On the computer, an input step of accepting a voice input related to a speech from the user; a classification step of classifying the utterance into which of the standard opening phrases should be returned, and predicting the standard opening phrase of the response to the utterance according to the classification result; a first output step of reading out voice data corresponding to the standard phrases at the beginning of the response predicted by the classification step from a voice data storage unit that stores voice data of a predetermined number of standard phrases used as the beginning of the response in advance, and outputting voice based on the read voice data; a processing step of sending a prompt including data of the utterance from the user to a natural language processing system utilizing a large-scale language model, and receiving a response from the natural language processing system; a speech synthesis step of synthesizing a speech related to the response received from the natural language processing system and generating the synthesized speech data following the speech data of the template; a second output step of outputting a voice based on the voice data synthesized in the voice synthesis step. [Explanation of symbols]

[0145] 100, 200...Robot system, 10...Conversational robot, 20. Working robot, 11, 21. Control device, 13. Microphone, 14···Speaker, 50···Natural language processing system, 111···Processing unit, 112, 212... Memory unit, 111a... Classifier, 111b... Search unit, 112a: Audio data storage unit; 112b: Input buffer; 113: Communication unit; 114: a voice input unit, 115: a voice recognition unit, 116: a voice synthesis unit, 117 Audio output section

Claims

1. A control device (11) for controlling a conversation with a user, an input unit (114) that accepts voice input related to speech from the user; an output unit (117) that outputs a voice related to a response to the utterance to the user; a voice data storage unit (112a) that stores in advance voice data of a predetermined number of standard phrases used as openings of responses; a processing unit (111) that transmits a prompt including data of the utterance from the user to a natural language processing system (50) that uses a large-scale language model, and receives a response from the natural language processing system; a speech synthesis unit (116) that performs speech synthesis based on a response from the natural language processing system; and The processing unit a classifier (111a) that classifies which of the standard opening phrases of the response should be returned in response to the utterance, and predicts the standard opening phrase of the response to the utterance in accordance with the classification result; reads out from the voice data storage unit voice data corresponding to the opening template of the response predicted by the classifier, and outputs voice to the output unit based on the read out voice data; a control device that causes the speech synthesis unit to synthesize speech related to the response received from the natural language processing system and generate synthesized speech data following the speech data of the template.

2. The processing unit a search unit (111b) for searching the voice data storage unit for voice data corresponding to the predicted standard phrase at the beginning of the response; The processing unit 2. The control device according to claim 1, wherein when voice data corresponding to the opening phrase of the response is found in the voice data storage unit through a search by the search unit, the voice corresponding to the opening phrase of the response is output to the output unit.

3. The processing unit 3. The control device according to claim 2, wherein, when the search unit does not find voice data corresponding to the opening template of the response in the voice data storage unit through a search, the control device causes the voice synthesis unit to generate voice data corresponding to the opening template of the response.

4. the response received from the natural language processing system is delimited by punctuation; The processing unit causing the search unit to search within the voice data storage unit for each punctuation mark unit separated by a punctuation mark in the answer; When the search unit finds the voice data corresponding to the punctuation unit in the voice data storage unit through the search, the output unit outputs the voice corresponding to the punctuation unit; 3. The control device according to claim 2, wherein when the search unit does not find voice data corresponding to the punctuation unit in the voice data storage unit through a search, the control device causes the voice synthesis unit to generate voice data corresponding to the punctuation unit.

5. The control device according to claim 3 , wherein the processing unit stores the voice data generated by the voice synthesis unit in the voice data storage unit.

6. The control device according to claim 4 , wherein the processing unit stores the voice data generated by the voice synthesis unit in the voice data storage unit.

7. The control device according to claim 1 , wherein the processing unit transmits the prompt to the natural language processing system, the prompt including an instruction to create a continuation of the standard opening phrase of the response predicted by the classifier.

8. 3. The control device according to claim 2, wherein when voice data corresponding to the opening template of the response is found in the voice data storage unit through a search by the search unit, the processing unit inserts a predetermined length of pause before outputting the voice corresponding to the opening template of the response to the output unit, and then outputs the voice corresponding to the opening template of the response to the output unit.

9. the voice data storage unit stores voice data of a standard phrase at the beginning of the response in association with a hash value, The control device according to claim 2 , wherein the search unit searches the voice data storage unit for the voice data using the hash value.

10. the processing unit stores the voice data generated by the voice synthesis unit in association with a hash value generated from a sentence that is the source of the voice data; The control device according to claim 2 , wherein the search unit searches the voice data storage unit for the voice data using the hash value.

11. The control device according to claim 1 , wherein the processing unit transmits the prompt to the natural language processing system, the prompt including an instruction for the natural language processing system to return the answer having a number of characters in one sentence not exceeding a predetermined number.

12. A control device (11) according to any one of claims 1 to 11; A robot system (100) having the natural language processing system (50).

13. A control device (21) for controlling a robot based on an interaction with a user, comprising: an input unit (114) that receives instructions from the user for the robot; an output unit (117) that outputs a voice related to a response to the instruction to the user; a voice data storage unit (112a) that stores in advance voice data of a predetermined number of standard phrases used as openings of responses; an input buffer (112b) for storing the instruction data input from the input unit; a processing unit (111) that sends a prompt including data read from the input buffer to a natural language processing system that uses a large-scale language model, and receives a response from the natural language processing system; a speech synthesis unit (116) that performs speech synthesis based on a response from the natural language processing system; and The processing unit a classifier (111a) for classifying the data read from the input buffer into which of the standard opening phrases of the response should be returned, and predicting the standard opening phrase of the response to the instruction in accordance with the classification result; reads out from the voice data storage unit voice data corresponding to the opening template of the response predicted by the classifier, and outputs voice to the output unit based on the read out voice data; a control device that causes the speech synthesis unit to synthesize speech related to the response received from the natural language processing system and generate synthesized speech data following the speech data of the template.

14. 1. A control method for controlling a conversation with a user, comprising: an input step of accepting a voice input related to a speech from the user; a classification step of classifying the utterance into which of the standard opening phrases should be returned, and predicting the standard opening phrase of the response to the utterance according to the classification result; a first output step of reading out voice data corresponding to the standard phrases for the opening of the response predicted by the classification step from a voice data storage unit that pre-stores voice data of a predetermined number of standard phrases used as openings of the response, and outputting voice based on the read voice data; a processing step of sending a prompt including data of the utterance from the user to a natural language processing system utilizing a large-scale language model, and receiving a response from the natural language processing system; a speech synthesis step of synthesizing a speech related to the response received from the natural language processing system and generating the synthesized speech data following the speech data of the template; a second output step of outputting a voice based on the voice data synthesized in the voice synthesis step.

15. A control program for controlling a conversation with a user, On the computer, an input step of accepting a voice input related to a speech from the user; a classification step of classifying the utterance into which of the standard opening phrases should be returned, and predicting the standard opening phrase of the response to the utterance according to the classification result; a first output step of reading out voice data corresponding to the standard phrases for the opening of the response predicted by the classification step from a voice data storage unit that pre-stores voice data of a predetermined number of standard phrases used as openings of the response, and outputting voice based on the read voice data; a processing step of sending a prompt including data of the utterance from the user to a natural language processing system utilizing a large-scale language model, and receiving a response from the natural language processing system; a speech synthesis step of synthesizing a speech related to the response received from the natural language processing system and generating the synthesized speech data following the speech data of the template; a second output step of outputting a voice based on the voice data synthesized in the voice synthesis step.

Images