system

Abstract

Provide a system. [Solution] A means for collecting user utterances as voice data; A means for converting the collected voice data into text data; A means for analyzing the converted text data and understanding the user's intent; means for generating a response text based on the analysis result; means for converting the generated response text into voice data; The system includes means for outputting the converted audio data to a user.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] "Problems to be solved" and "Means to solve the problems"

[0005] Conventional conversational robots rely on preset patterns and operator responses, making it difficult to achieve flexible and natural dialogue. Furthermore, existing systems lack the accuracy of information gathering and response generation, limiting the user experience. The present invention aims to realize automated conversations using speech generation AI, providing more natural and flexible dialogue. In particular, the present invention aims to improve accuracy and user experience by providing a system that accurately understands user utterances and provides appropriate responses. [Means for solving the problem]

[0006] The present invention provides an automated conversation system that includes a means for collecting user utterances as voice data, a means for converting the collected voice data into text data, a means for analyzing the converted text data to understand the user's intent, a means for generating a response text based on the analysis results, a means for converting the generated response text into voice data, and a means for outputting the converted voice data to the user. The system also includes a means for receiving feedback from the user and using it to learn and improve the system, thereby continuously improving the accuracy and usability of the system. Furthermore, the system also includes a means for acquiring external information based on the analysis results and a means for generating a response text based on the acquired external information, making it possible to provide highly accurate answers based on specific information.

[0007] "User" refers to a person who interacts with a conversational robot.

[0008] "Utterance" refers to the sound spoken by the user.

[0009] "Voice data" refers to data that represents in digital form a user's speech captured through an input device such as a microphone.

[0010] "Text data" refers to data in the form of a string of characters that has been converted from voice data using voice recognition technology.

[0011] A "voice recognition engine" refers to a software or hardware system for converting voice data into text data.

[0012] A "natural language processing (NLP) engine" refers to technology for analyzing text data and understanding its meaning and intent.

[0013] "Response text" refers to data that expresses the response content generated based on the user's utterance and intention in the form of a string.

[0014] "Voice generation AI" refers to artificial intelligence that generates response text using natural language processing and machine learning technology.

[0015] "TTS (Text-to-Speech) engine" refers to the technology that converts text data into voice data.

[0016] "External information" refers to various data obtained from outside the system, such as weather information.

[0017] "Feedback" refers to the evaluation and suggestions for correction of the system's response provided by the user.

[0018] The term "system" refers to a set of hardware and software that includes the above means and realizes automated conversation. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] Detailed Description of the Invention

[0041] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[0042] The system mainly consists of the following components:

[0043] 1. Voice input means (terminal)

[0044] 2. Speech recognition means (server)

[0045] 3. Natural language processing means (server)

[0046] 4. Response Generation Method (Server)

[0047] 5. Speech synthesis means (server)

[0048] 6. Audio output means (terminal)

[0049] 7. Feedback Collection Method (Server)

[0050] A means of collecting user utterances

[0051] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[0052] Converting audio data to text

[0053] The server converts the collected voice data into text data using a voice recognition engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[0054] Analysis of speech content

[0055] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes the intent as "ask about the current temperature."

[0056] Generate a response

[0057] Based on the analysis results, the server uses speech generation AI to generate an appropriate response, such as "The current temperature is 25 degrees."

[0058] Response text transcription

[0059] The server converts the generated response text into voice data using a speech synthesis engine, which generates a voice that the user perceives as natural speech.

[0060] Audio output

[0061] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[0062] Gathering feedback and learning

[0063] If the user's response is inaccurate or inappropriate, they can make corrections or additions. This feedback is sent to the server and added to the system's learning database. The server uses the feedback to retrain the system to improve its performance.

[0064] Specific examples

[0065] The process will be explained as a concrete example when a user asks a robot, "What's the temperature today?"

[0066] 1. User says: "What time is it today?"

[0067] 2. The device's microphone captures the audio and sends the audio data to the server.

[0068] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[0069] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[0070] 5. The server uses a speech generation AI to generate a response text saying, "The current temperature is 25 degrees."

[0071] 6. The server uses a speech synthesis engine to convert the response text into voice data.

[0072] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[0073] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[0074] 9. The server receives the feedback and retrains the system.

[0075] This will enable natural and fluent dialogue between the robot and the user, resulting in a system that improves the user experience.

[0076] The processing flow will be explained below.

[0077] Specific processing steps of the program

[0078] Step 1:

[0079] The user speaks to the robot. At this time, the user's utterance is input as voice data, such as "What is the temperature today?"

[0080] Step 2:

[0081] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[0082] Step 3:

[0083] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[0084] Step 4:

[0085] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the user's intent: "I'm asking about the current temperature."

[0086] Step 5:

[0087] Based on the analysis results, the server calls a response generation module, which sends a request to an external API that provides temperature information. For example, the API returns the data "25 degrees."

[0088] Step 6:

[0089] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text. The generated response text is "The current temperature is 25 degrees."

[0090] Step 7:

[0091] The server inputs the generated response text into a text-to-speech engine (TTS), which converts the text data into speech data, generating speech that is easy for the user to understand.

[0092] Step 8:

[0093] The server sends the converted audio data to the device, which then plays it back through the device's speaker.

[0094] Step 9:

[0095] The user hears the robot's response. This time, the user receives the response, "The current temperature is 25 degrees."

[0096] Step 10:

[0097] If the user feels that the answer is not accurate, they can ask questions again or provide additional feedback. This feedback information is again collected by the device's microphone and sent to the server.

[0098] Step 11:

[0099] The server receives user feedback and adds it to the system's learning database, which allows the system to retrain and improve the accuracy of future responses.

[0100] This completes the process of an automated conversation system that utilizes voice generation AI.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Conventional automated conversation systems have difficulty accurately understanding user intent and generating appropriate responses. Furthermore, they lack the ability to incorporate user feedback into learning to improve the system's accuracy, resulting in insufficient improvement in the user experience. Furthermore, the system's ability to effectively obtain external information and generate responses based on it is also limited. This leaves users unable to have satisfying conversations.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating response text using a generative AI model based on the analysis results, and means for converting the generated response text into voice data. This enables the server to generate appropriate and natural responses to user questions and improve the user experience. Furthermore, by including means for receiving feedback from the user and using it to learn and improve the system, the accuracy of the system can be improved. Furthermore, by including means for acquiring external information based on the analysis results, the server can provide more comprehensive responses.

[0106] The "means for collecting user speech as voice data" refers to a device or method for inputting the user's speech as a digital signal.

[0107] The "means for converting collected voice data into text data" refers to a technique or device that analyzes collected voice data and converts it into corresponding text data.

[0108] The "means for analyzing the converted text data and understanding the user's intent" refers to a technique or method for analyzing the text data using natural language processing technology and identifying the user's speech intent.

[0109] "Means for generating response text using a generative AI model based on analysis results" refers to techniques and methods that utilize a generative AI model to create appropriate response text based on analysis results that understand the user's intent.

[0110] The "means for converting the generated response text into voice data" refers to a technique or device that converts the generated response in text format into voice data using voice synthesis technology.

[0111] "Means for receiving user feedback and using it to learn and improve the system" refers to techniques and methods for collecting feedback information provided by users and incorporating it into the re-learning process using machine learning algorithms to improve the system and increase its accuracy.

[0112] "Means for acquiring external information based on the analysis results" refers to techniques or methods for acquiring the necessary external information from external sources such as databases or the Internet in accordance with the analyzed user's intentions.

[0113] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[0114] Voice input means

[0115] When a user speaks to the robot, their voice is collected by the device's microphone. The collected voice data is sent to the server in real time. Examples of voice input methods include a smartphone or a dedicated voice input device.

[0116] Converting audio data to text

[0117] The server that receives the voice data converts the collected voice data into text data using a speech recognition engine such as the Google® Speech-to-Text API. For example, a user's utterance, "What time is it today?", is generated as text data. This process includes feature extraction and phoneme recognition technologies to improve the accuracy of the voice-to-text conversion.

[0118] Analysis of speech content

[0119] The converted text data is sent to the server's natural language processing (NLP) engine. NLP engines such as SpaCy and Google Cloud Natural Language API are used. This analyzes the text data and understands the user's intent. At this point, the system recognizes the intent to "ask about the current temperature."

[0120] Generate a response

[0121] Based on the analysis results, the server uses a generative AI model (e.g., OpenAI (registered trademark) GPT-4 (registered trademark)) to generate an appropriate response text. For example, a response text such as "The current temperature is 25 degrees" is generated. A specific example of an input prompt sentence for the generative AI model is as follows:

[0122] "The user asked 'What's the temperature today?' Generate a response."

[0123] Response text transcription

[0124] The generated response text is converted into voice data using a speech synthesis engine such as the Google Text-to-Speech API on the server. The speech synthesis engine has phonetic conversion technology and a speech waveform generation algorithm for converting text into speech waveforms.

[0125] Audio output

[0126] The generated voice data is sent from the server to the device. It is played back as voice from the device's speaker, and the user can hear the reply, "The current temperature is 25 degrees." The voice output means can be the robot's speaker or a smartphone speaker.

[0127] Gathering feedback and learning

[0128] If a user's response is incorrect or inappropriate, they can provide corrections or additional feedback. This feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain the system to improve its performance. A machine learning framework such as TENSORFLOW® is used for retraining.

[0129] Specific example explanation

[0130] As a concrete example, if a user asks a robot "What time is it today?", the process is as follows:

[0131] 1. User says: "What time is it today?"

[0132] 2. The device's microphone captures the audio and sends the audio data to the server.

[0133] 3. The server uses the Google Speech-to-Text API to convert the voice data into text: "What time is it today?"

[0134] 4. The server uses an NLP engine to parse the text and understand the intent: asking about the current temperature.

[0135] 5. The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[0136] 6. The server uses the Google Text-to-Speech API to convert the response text into audio data.

[0137] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[0138] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[0139] 9. The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[0140] This series of processes enables the user to have natural and fluent conversations with the robot, resulting in a system that improves the user experience.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] User says: "What time is it today?"

[0144] Input: User's spoken utterance

[0145] Output: Digital audio data captured by the device's microphone

[0146] How it works: The user speaks to the robot, and their voice is captured through the device's microphone. At this stage, the voice is temporarily stored as digital data in a buffer.

[0147] Step 2:

[0148] The device's microphone captures the audio and sends the audio data to the server.

[0149] Input: Digital audio data

[0150] Output: Digital audio data sent to the server

[0151] Operation: The terminal transmits the captured audio data to the server in real time. The audio data is transmitted over the network using the TCP / IP protocol.

[0152] Step 3:

[0153] The server converts the audio data into text using the Google Speech-to-Text API.

[0154] Input: Digital audio data sent to the server

[0155] Output: Text data "What's the temperature today?"

[0156] How it works: The server passes the received audio data to the Google Speech-to-Text API, which converts the audio data into phonemes and generates the corresponding text data based on them.

[0157] Step 4:

[0158] The server uses an NLP engine to parse the text and understand the intent of asking about the current temperature.

[0159] Input: Text data "What's the temperature today?"

[0160] Output: Parsing the intent "Ask about the current temperature"

[0161] How it works: The text data is sent to the server's NLP engine (e.g., SpaCy or Google Cloud Natural Language API), which uses intent recognition algorithms to analyze the user's question intent.

[0162] Step 5:

[0163] The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[0164] Input: Parsed intent "Ask about the current temperature", prompt: "The user asked 'What's the temperature today?' Generate a response."

[0165] Output: Response text "The current temperature is 25 degrees."

[0166] How it works: The server provides the generative AI model with an input prompt and generates an appropriate response text based on the intent. The generative AI model creates this response based on a pre-trained dataset.

[0167] Step 6:

[0168] The server converts the response text into audio data using the Google Text-to-Speech API.

[0169] Input: Response text "The current temperature is 25 degrees"

[0170] Output: Audio data saying "The current temperature is 25 degrees"

[0171] How it works: The generated response text is sent to the server's Google Text-to-Speech API, where phonetic conversion from text to speech and generation of speech waveforms are performed, resulting in audio data.

[0172] Step 7:

[0173] The device plays the generated audio data over the speaker: "The current temperature is 25 degrees."

[0174] Input: Speech data saying "The current temperature is 25 degrees"

[0175] Output: Voice played through speaker: "The current temperature is 25 degrees."

[0176] Operation: Audio data is sent from the server to the device and output to the user as audio through the device's speaker.

[0177] Step 8:

[0178] The user listens to the response and may ask further questions or provide additional feedback.

[0179] Input: "The current temperature is 25 degrees"

[0180] Output: User feedback

[0181] How it works: The user listens to the reply and, if necessary, enters corrections or additional feedback into the terminal, which is then sent to the server.

[0182] Step 9:

[0183] The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[0184] Input: Feedback data received from users

[0185] Output: Improved model

[0186] How it works: Feedback data is stored on a server and the model is retrained using a machine learning framework such as TensorFlow, improving the accuracy of the system and the user experience.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Conventional autonomous vehicles have difficulty in naturally interacting with users, requiring drivers and passengers to perform complex operations to set destinations, obtain current location information, and receive advice on safe driving. Furthermore, there has been a lack of easy ways to obtain external information in real time, such as weather information notifications, which has hindered the user experience of autonomous driving.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating a response text based on the analysis result, means for converting the generated response text into voice data, means for outputting the converted voice data to the user, and means for functioning as a voice assistant in an autonomous vehicle and interactively setting a destination, providing current location information, notifying weather information, and providing advice on safe driving. This enables multifunctional interaction through natural conversation with the user in an autonomous vehicle.

[0192] A "means for collecting user speech as audio data" is a device or method that captures the user's speech and stores it as digital audio data.

[0193] The "means for converting collected voice data into text data" refers to a process of converting voice data into corresponding character strings using voice recognition technology.

[0194] "Means for analyzing the converted text data and understanding the user's intent" refers to a function that uses natural language processing technology to analyze the text data and understand the user's intent and questions from its content.

[0195] The "means for generating a response text based on the analysis results" is a process for generating an appropriate response content in response to the analyzed user intent.

[0196] The "means for converting the generated response text into voice data" is a voice synthesis technology for generating natural and fluent voice from text.

[0197] The "means for outputting the converted voice data to the user" refers to a device or method for making the generated voice data audible to the user through a speaker or the like.

[0198] "A means that functions as a voice assistant within an autonomous vehicle and interactively provides destination setting, current location information, weather information, and advice on safe driving" refers to a system or software that can interact with the user via voice within the environment of the autonomous vehicle and respond to various information and instructions.

[0199] System Program Overview

[0200] A system for implementing this invention includes the following major components:

[0201] 1. Voice input means (terminal)

[0202] 2. Speech recognition means (server)

[0203] 3. Natural language processing means (server)

[0204] 4. Response Generation Method (Server)

[0205] 5. Speech synthesis means (server)

[0206] 6. Audio output means (terminal)

[0207] 7. Feedback Collection Method (Server)

[0208] Hardware and Software

[0209] The system uses a terminal equipped with a microphone and speaker to collect the user's speech in real time. The server uses the following software to process and calculate the data:

[0210] Speech recognition: Use the speech_recognition library to convert voice data into text data.

[0211] Natural language processing and response generation: Uses OpenAI's API to analyze user intent and generate appropriate response text.

[0212] Speech synthesis: Use the gtts library to convert response text into voice data.

[0213] Audio output: The generated audio data is played through the speaker using the playsound library.

[0214] Feedback collection: Receive feedback from users and use it to train the system.

[0215] Specific examples

[0216] User questions and how to handle them

[0217] For example, consider the case where a user says, "How much battery power is currently remaining in my vehicle?" The device's microphone captures the voice and sends the voice data to the server. The server then uses a speech recognition engine to convert this voice data into text. The server then analyzes this text with a natural language processing engine to understand the intent of the user's question. Based on the analysis results, a response generation engine generates a response text such as "The current battery power remaining is 80%." The generated response text is then converted into voice data using a speech synthesis engine and sent to the device. The device's speaker plays this voice data, conveying the response to the user.

[0218] Prompt Sentence Examples

[0219] Specific prompts are used to guide the generative AI model to generate appropriate responses. For example, here are some examples of prompts that can be used when a user asks:

[0220] Answer the following question: "How much battery power is left in my current vehicle?"

[0221] Based on this prompt, the system generates an appropriate response and outputs it as voice data.

[0222] Feedback collection and learning

[0223] If a user provides feedback on a response, that feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve the system's performance. For example, if a user provides feedback indicating dissatisfaction with a response, the system will adjust to provide better responses to similar questions in the future.

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] A means of collecting user utterances

[0227] When a user speaks in an autonomous vehicle, the device's microphone captures the voice. This voice data is collected in digital form and sent to a server. The input is the user's voice, and the output is digital voice data.

[0228] Step 2:

[0229] A means of converting audio data into text data

[0230] The server converts the received voice data into text data using a speech recognition engine (speech_recognition library). In this process, the voice waveform data is analyzed and the corresponding text is generated. The input is digital voice data, and the output is text data.

[0231] Step 3:

[0232] A means of analyzing text data and understanding user intent

[0233] The server analyzes the converted text data using a natural language processing engine (OpenAI API) to understand the user's intent. This analysis identifies what the user is looking for based on the content of the input text. The input is text data, and the output is the user's intent (for example, checking the remaining battery level).

[0234] Step 4:

[0235] A way to generate response text

[0236] Based on the analysis results, the server uses a response generation engine (OpenAI API) to generate an appropriate response text. The generative AI model creates a response text based on the prompt text. The input is the user's intention, and the output is the response text. An example of a specific prompt text is "Please answer the following question: 'How much battery is left in your current vehicle?'"

[0237] Step 5:

[0238] A means of converting response text into audio data

[0239] The server converts the generated response text into voice data using a speech synthesis engine (gtts library). Natural and fluent speech is generated from the text. The input is the response text, and the output is voice data.

[0240] Step 6:

[0241] A means for outputting audio data to the user

[0242] The server sends the generated voice data to the device and plays it from the device's speaker, allowing the user to hear the reply. The input is voice data, and the output is voice played from the speaker.

[0243] Step 7:

[0244] Gathering feedback and learning

[0245] Users can provide feedback on responses. For example, if a response is inappropriate, the user can offer corrections or suggestions for clarification. This feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve system performance. The input is the user's feedback, and the output is an updated learning database.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] Detailed Description of the Invention

[0248] The present invention provides an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. This system not only collects and analyzes user utterances to generate responses, but also recognizes the user's emotions and responds based on those emotions. Specific embodiments of the system are described below.

[0249] The system mainly consists of the following components:

[0250] 1. Voice input means (terminal)

[0251] 2. Speech recognition means (server)

[0252] 3. Natural language processing means (server)

[0253] 4. Emotion Recognition Method (Server)

[0254] 5. Response Generation Method (Server)

[0255] 6. Speech synthesis means (server)

[0256] 7. Audio output means (terminal)

[0257] 8. Feedback Collection Method (Server)

[0258] A means of collecting user utterances

[0259] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[0260] Converting audio data to text

[0261] The server converts the collected voice data into text data using an automatic speech recognition (ASR) engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[0262] Analysis of speech content

[0263] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes that the user is asking about the current temperature.

[0264] Emotion recognition

[0265] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from their speech. For example, the utterance "What's the temperature today?" is recognized as "excited" or "calm."

[0266] Generate a response

[0267] Based on the analysis results and emotion recognition results, the server uses speech generation AI to generate an appropriate response text. For example, it might generate a response such as "The current temperature is 25 degrees." The response text is adjusted according to the user's emotions. If the user is excited, the response will be adjusted to sound calm.

[0268] Response text transcription

[0269] The server converts the generated response text into voice data using a text-to-speech engine (TTS). At this time, the tone and speed of the voice are adjusted based on the user's emotions. For example, if the user is calm, the response will be in a calm tone.

[0270] Audio output

[0271] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the response adjusted with the appropriate tone and speed.

[0272] Gathering feedback and learning

[0273] If the user's response is incorrect or inappropriate, they can make corrections or clarifications. This feedback is collected by the device's microphone and sent back to the server.

[0274] The server receives the feedback and adds it to the system's learning database, which allows the system to retrain itself and improve the accuracy of future responses.

[0275] Specific examples

[0276] Let us take a concrete example of the process when a user asks the robot in an excited voice, "What's the temperature today?"

[0277] 1. User says: "What's the temperature today?" (in an excited voice)

[0278] 2. The device's microphone captures the audio and sends the audio data to the server.

[0279] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[0280] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[0281] 5. The server uses an emotion recognition engine to recognize the emotion of the utterance as "excited."

[0282] 6. The server calls the response generation module based on the analysis results and emotion recognition results, and the speech generation AI generates the response text, "The current temperature is 25 degrees."

[0283] 7. The server converts the response text into voice data using a speech synthesis engine, responding in a calm tone to an excited user.

[0284] 8. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[0285] 9. The user listens to the response and possibly asks further questions or provides additional feedback.

[0286] 10. The server receives user feedback and retrains the system.

[0287] This will enable automated conversation systems that utilize voice generation AI and emotion recognition engines to achieve more natural and flexible dialogue, improving the user experience.

[0288] The processing flow will be explained below.

[0289] Detailed Description of the Invention

[0290] The specific processing steps of a conversation system incorporating an emotion recognition engine will be described below.

[0291] Step 1:

[0292] The user speaks to the robot, for example, saying, "What's the temperature today?"

[0293] Step 2:

[0294] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[0295] Step 3:

[0296] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[0297] Step 4:

[0298] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the intent that the user is asking about the current temperature.

[0299] Step 5:

[0300] At the same time, the server passes the voice data to an emotion recognition engine to analyze the user's emotions, which then recognizes the user's speech as "excited" or "calm."

[0301] Step 6:

[0302] Based on the analysis results (intent and emotion), the server calls the response generation module. This module sends a request to an external API to obtain temperature information. For example, the API returns the data "25 degrees."

[0303] Step 7:

[0304] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text, which is "The current temperature is 25 degrees."

[0305] Step 8:

[0306] The server inputs the response text into a text-to-speech engine (TTS) and converts the text data into voice data. During voice synthesis, the tone and speed of the voice are adjusted according to the user's emotions. For example, if the user is excited, the server will respond in a calm tone.

[0307] Step 9:

[0308] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[0309] Step 10:

[0310] If the user feels that the answer is not accurate, they can ask the question again or provide additional feedback, which is again collected by the device's microphone and sent to the server.

[0311] Step 11:

[0312] The server receives user feedback and adds it to the system's learning database, allowing the system to retrain based on the feedback and improve the accuracy of future responses.

[0313] This enables speech generation AI systems incorporating emotion recognition engines to enable more natural and emotionally appropriate dialogue, improving the user experience.

[0314] Example 2

[0315] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] Conventional automated conversation systems can collect user utterances, convert them into text data, and generate responses. However, they are unable to interpret the user's emotions and respond appropriately, resulting in a lack of naturalness in the dialogue and a poor user experience. Furthermore, it is difficult to utilize feedback to continuously improve the system and respond to diverse user needs.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0318] In this invention, the server includes means for recognizing emotions from user utterances, means for generating response text based on the analysis results and emotion recognition results, and means for receiving feedback from the user and using it to learn and improve the system, thereby enabling more natural and effective dialogue that takes user emotions into consideration and allowing the system to learn and improve accuracy.

[0319] "User utterance" refers to a linguistic expression that a user verbally inputs to the system.

[0320] "Voice data" refers to the user's speech converted into a digital signal.

[0321] A "collection means" is a device or program that captures a user's voice data and stores it for processing.

[0322] "Text data" refers to character string information obtained by analyzing voice data.

[0323] "Means for converting" refers to devices or software for converting voice data into text data.

[0324] "Means of analysis" refers to machine learning models and programs that break down and analyze text data to understand its content and intent.

[0325] "User intent" refers to the information or request that a user wants to convey through speech.

[0326] "Means for recognizing emotions" refers to algorithms or programs that determine the emotions expressed by the user at that time from their speech.

[0327] "Analysis results" refers to the information obtained after interpreting the content and intent of text data.

[0328] The "emotion recognition result" is the classification result of emotions analyzed from the user's speech.

[0329] "Response text" is a character string representing the response content corresponding to the user's utterance.

[0330] "Generating means" refers to a device or program that creates a response text based on the analysis results and emotion recognition results.

[0331] "Means for converting into voice data" refers to the device or software that converts the response text back into voice form.

[0332] "Output means" refers to a device or program that plays the converted audio data on a speaker or the like so that the user can hear it.

[0333] "Feedback" refers to the evaluation or opinion that a user gives to the system's response.

[0334] "Means used to learn and improve the system" refers to machine learning algorithms and programs that incorporate user feedback to improve the system's performance and response accuracy.

[0335] This invention provides an automatic conversation system that utilizes a voice generation AI and an emotion recognition engine. Specifically, it is designed not only to collect user utterances, analyze them, and generate responses, but also to recognize the user's emotions and respond based on those emotions.

[0336] The system includes the following components:

[0337] 1. Voice input means (terminal)

[0338] 2. Speech recognition means (server)

[0339] 3. Natural language processing means (server)

[0340] 4. Emotion Recognition Method (Server)

[0341] 5. Response Generation Method (Server)

[0342] 6. Speech synthesis means (server)

[0343] 7. Audio output means (terminal)

[0344] 8. Feedback Collection Method (Server)

[0345] The device is equipped with a microphone that collects the user's voice. This microphone captures the voice spoken by the user and converts it into voice data. For example, if a user speaks to the robot, "What time is it today?", the voice is collected by the microphone.

[0346] The device sends the collected voice data to the server in real time. A secure protocol (e.g., HTTPS) is used for this transmission. The server then converts the received voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). For example, the voice "What's the temperature today?" is output as the text "What's the temperature today?"

[0347] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., GPT-3 (registered trademark)). During this process, the server understands that the intent of the text content is "asking about the current temperature." At the same time, the server uses an emotion recognition engine (e.g., IBM Watson (registered trademark) Tone Analyzer) to analyze the emotion from the user's utterance. For example, the utterance "What's the temperature today?" is recognized as "excited."

[0348] Based on the analysis results and emotion recognition results, the server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text. For example, the response text generated might be, "The current temperature is 25 degrees." The tone of the response is adjusted according to the user's emotion. If the user is excited, the response will be set to a calm tone.

[0349] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Amazon Polly). The tone and speed of the voice are also adjusted based on the user's emotions. For example, if the response is made in a calm tone, the server will respond in a calm tone, saying, "The current temperature is 25 degrees."

[0350] The terminal receives the generated voice data and plays it back through the speaker. The user can hear the response, adjusted with the appropriate tone and speed. An example of a specific prompt sentence is the following dialogue when the user asks, "What time is it today?"

[0351] If the user adds or corrects a reply, that feedback is collected by the device's microphone and sent back to the server. The server receives the feedback, adds it to the system's learning database, and retrains it to improve the accuracy of future replies. This allows the system to continuously improve and provide more natural and accurate dialogue.

[0352] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0353] Step 1:

[0354] The user says, "What's the temperature today?"

[0355] Input: User's speech (voice data)

[0356] Output: Audio data captured by the device's microphone

[0357] Specific operation: The device's microphone captures the user's voice and obtains the voice data.

[0358] Step 2:

[0359] The device transmits the audio data to the server in real time.

[0360] Input: Audio data captured by the device's microphone

[0361] Output: Audio data sent to the server

[0362] What happens: The device sends the captured audio data to the server via a secure protocol (e.g., HTTPS).

[0363] Step 3:

[0364] The server converts the voice data into text data using a voice recognition engine.

[0365] Input: Transmitted audio data

[0366] Output: The converted text data (e.g. "What time is it today?")

[0367] Specific operation: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data.

[0368] Step 4:

[0369] The server uses a natural language processing engine to analyze the text data and understand the user's intent.

[0370] Input: Converted text data

[0371] Output: Analysis results indicating the user's intent (e.g., "I'm asking for the current temperature")

[0372] Specific operation: The server uses a natural language processing (NLP) engine (e.g., GPT-3) to analyze text data and understand the user's intent.

[0373] Step 5:

[0374] The server uses an emotion recognition engine to recognize emotions from the user's speech.

[0375] Input: Converted text data

[0376] Output: Recognition result indicating the user's emotion (e.g. excited)

[0377] Specific operation: The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze emotions from the user's speech.

[0378] Step 6:

[0379] The server generates a response text based on the analysis results and emotion recognition results.

[0380] Input: Analysis results and emotion recognition results that indicate the user's intentions

[0381] Output: Generated response text (e.g. "The current temperature is 25 degrees")

[0382] Specific operation: The server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text.

[0383] Step 7:

[0384] The server converts the generated response text into audio data.

[0385] Input: Generated response text

[0386] Output: Converted audio data

[0387] What happens: The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the response text into speech data, adjusting the tone and speed of the speech.

[0388] Step 8:

[0389] The server sends the audio data to the terminal.

[0390] Input: Converted audio data

[0391] Output: Audio data sent to the device

[0392] Specific operation: The server sends the audio data to the device using a secure protocol (e.g. HTTPS).

[0393] Step 9:

[0394] The audio data received by the device is output from the speaker.

[0395] Input: Transmitted audio data

[0396] Output: What the user hears (e.g., "The current temperature is 25 degrees")

[0397] What happens: The response is played through the device speaker with a controlled voice tone and speed.

[0398] Step 10:

[0399] The user provides feedback on the response.

[0400] Input: User feedback (voice data)

[0401] Output: Feedback audio data captured by the device's microphone

[0402] Specific operation: The device's microphone captures the user's feedback and obtains the audio data.

[0403] Step 11:

[0404] The terminal transmits the feedback voice data to the server.

[0405] Input: Captured feedback audio data

[0406] Output: Feedback audio data sent to the server

[0407] Specific operation: The device sends feedback audio data to the server via a secure protocol (e.g., HTTPS).

[0408] Step 12:

[0409] The server receives the feedback, adds it to the system's learning database, and retrains it.

[0410] Input: Submitted feedback audio data

[0411] Output: Updated training database and improved system

[0412] Specific operation: The server analyzes the feedback, adds it to the system's learning database, and performs re-training to improve response accuracy.

[0413] (Application example 2)

[0414] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0415] Conventional automated conversation systems can analyze user utterances and generate responses, but they face the challenge of generating appropriate responses that take the user's emotions into account. Furthermore, for users to safely and naturally converse in an autonomous vehicle while driving or traveling, responses that reflect the user's emotional state are required. This calls for a more advanced dialogue system.

[0416] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intention, means for recognizing emotions from the user's utterance, means for generating a response text based on the analysis result and the emotion recognition result, means for converting the generated response text into voice data, and means for outputting the converted voice data to the user. This enables an appropriate response according to the user's emotions.

[0417] "User" refers to a person who uses the system.

[0418] "Utterance" refers to words spoken by a user.

[0419] "Voice data" refers to data that is an electronic recording of a user's speech.

[0420] "Collection means" refers to a device or method for acquiring audio data.

[0421] "Text data" refers to data that has been converted from voice data into text information.

[0422] "Conversion means" refers to a device or method for converting audio data into text data.

[0423] "Analysis means" refers to a device or method that analyzes text data and understands the user's intent and content.

[0424] "Emotion recognition means" refers to a device or method for identifying emotions from a user's speech.

[0425] "Response generation means" refers to a device or method that generates an appropriate response based on the analysis results and emotion recognition results.

[0426] "Speech synthesis means" refers to a device or method that converts generated text responses into speech.

[0427] "Audio output means" refers to a device or method for reproducing audio data and allowing the user to listen to it.

[0428] "Feedback collection means" refers to a device or method for collecting corrections or supplemental information from users.

[0429] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[0430] This invention is an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. A specific embodiment for collecting user utterances in an autonomous vehicle and generating appropriate responses is described below.

[0431] System Overview

[0432] The system mainly consists of the following components:

[0433] 1. Voice input means (terminal)

[0434] 2. Speech recognition means (server)

[0435] 3. Natural language processing means (server)

[0436] 4. Emotion Recognition Method (Server)

[0437] 5. Response Generation Method (Server)

[0438] 6. Speech synthesis means (server)

[0439] 7. Audio output means (terminal)

[0440] 8. Feedback Collection Method (Server)

[0441] Voice input means

[0442] The device includes a microphone placed inside the autonomous vehicle to collect the user's voice, and the microphone is equipped with noise cancellation to ensure high-quality capture of the passenger's voice.

[0443] Voice recognition means

[0444] The server converts the collected voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API), thereby obtaining the user's speech as text data.

[0445] Natural language processing tools

[0446] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., Transformers Library). This analysis helps to understand the user's intent and the content of the question.

[0447] emotion recognition means

[0448] At the same time, the server uses an emotion recognition engine (e.g., Hugging Face emotion analysis model) to analyze the emotions from the user's speech. Emotion recognition identifies the user's state of mind based on the content of the text data.

[0449] Response generation method

[0450] The server uses a response generation AI (e.g., rinna / japanese-gpt2-medium) to generate appropriate response text based on the analysis and emotion recognition results. The generated response is tailored to the user's emotions.

[0451] Voice synthesis means

[0452] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Google Cloud Text-to-Speech API). During this conversion process, the voice tone and speed are adjusted according to the user's emotions.

[0453] Audio output means

[0454] The voice data is sent to the terminal and played back through the vehicle's speakers, allowing the user to receive a voice response.

[0455] Feedback collection methods

[0456] If a user's response is incorrect or inappropriate, they can make corrections or additions. This feedback is collected by the device's microphone and sent to the server, which receives the feedback and adds it to the system's learning database to improve the accuracy of the responses.

[0457] Specific examples

[0458] For example, if a user in a self-driving car asks "How long until the next service stop?" (in a worried voice), the system might:

[0459] 1. The device's microphone captures audio and sends the audio data to the server.

[0460] 2. The server uses a speech recognition engine to convert the voice data into text.

[0461] 3. The server analyzes the text using a natural language processing engine and understands the question.

[0462] 4. The server uses its emotion recognition engine to recognize the emotion of the utterance as "worry."

[0463] 5. The server uses a response generation AI to generate a response text that reads, "The next service area is about 30 minutes away. Please relax."

[0464] 6. When the server converts the response text into voice data using a speech synthesis engine, it responds in a gentle tone.

[0465] 7. The device plays the generated audio data from the speaker.

[0466] Prompt Sentence Examples

[0467] Input: "As a concierge AI for a self-driving vehicle, you are asked by a concerned passenger how long it will be to the next stop. Please create a response that will ease their concern."

[0468] Generate: "The next service area is about 30 minutes away. Relax."

[0469] This enables an automated conversation system that utilizes voice generation AI and an emotion recognition engine to provide flexible responses according to the user's emotional state, improving the user experience.

[0470] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0471] Step 1:

[0472] The device's microphone collects the user's speech and captures it as voice data. The input is the user's speech, and the output is voice data. This voice data is stored on the device as an electronic signal.

[0473] Step 2:

[0474] The device sends the collected voice data to the server. The input is the voice data stored on the device, and the output is the voice data transferred to the server.

[0475] Step 3:

[0476] The server uses a speech recognition engine to convert the voice data into text data. The input is voice data and the output is text data. The speech recognition engine (e.g., Google Speech-to-Text API) performs this conversion.

[0477] Step 4:

[0478] The server uses a natural language processing engine to analyze the text data and understand the user's intent. The input is text data, and the output is an analysis result that indicates the user's intent. The natural language processing engine (e.g., Transformers Library) performs this analysis.

[0479] Step 5:

[0480] The server uses an emotion recognition engine to recognize the user's emotions from the text data. The input is the text data, and the output is the emotion recognition result. The emotion recognition engine (e.g., Hugging Face emotion analysis model) performs this analysis.

[0481] Step 6:

[0482] The server generates a response text using a response generation AI based on the analysis results and emotion recognition results. The input is the analysis results and emotion recognition results, and the output is the response text. This generation is performed by the response generation AI (e.g., rinna / japanese-gpt2-medium).

[0483] Step 7:

[0484] The server uses a speech synthesis engine to convert the generated response text into audio data. The input is the response text and the output is audio data. The speech synthesis engine (e.g., Google Cloud Text-to-Speech API) performs this conversion.

[0485] Step 8:

[0486] The server sends the voice data to the terminal. The input is the voice data generated by the server, and the output is the voice data transferred to the terminal.

[0487] Step 9:

[0488] The device plays the received audio data from the speaker. The input is the audio data stored on the device, and the output is the audio played from the speaker.

[0489] Step 10:

[0490] The user hears the reply and provides feedback on the content. The input is the user's recognition and the output is the feedback speech.

[0491] Step 11:

[0492] The terminal captures the feedback voice from the user and sends it to the server. The input is the feedback voice, and the output is the feedback voice transferred to the server.

[0493] Step 12:

[0494] The server analyzes the feedback speech and uses it to improve the system. The input is the feedback speech, and the output is the system's learning and improvement results. This allows the system to generate more accurate responses.

[0495] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0496] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0497] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0498] [Second embodiment]

[0499] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0500] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0501] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0502] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0503] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0504] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0505] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0506] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0507] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0508] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0509] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0510] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0511] Detailed Description of the Invention

[0512] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[0513] The system mainly consists of the following components:

[0514] 1. Voice input means (terminal)

[0515] 2. Speech recognition means (server)

[0516] 3. Natural language processing means (server)

[0517] 4. Response Generation Method (Server)

[0518] 5. Speech synthesis means (server)

[0519] 6. Audio output means (terminal)

[0520] 7. Feedback Collection Method (Server)

[0521] A means of collecting user utterances

[0522] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[0523] Converting audio data to text

[0524] The server converts the collected voice data into text data using a voice recognition engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[0525] Analysis of speech content

[0526] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes the intent as "ask about the current temperature."

[0527] Generate a response

[0528] Based on the analysis results, the server uses speech generation AI to generate an appropriate response, such as "The current temperature is 25 degrees."

[0529] Response text transcription

[0530] The server converts the generated response text into voice data using a speech synthesis engine, which generates a voice that the user perceives as natural speech.

[0531] Audio output

[0532] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[0533] Gathering feedback and learning

[0534] If the user's response is inaccurate or inappropriate, they can make corrections or additions. This feedback is sent to the server and added to the system's learning database. The server uses the feedback to retrain the system to improve its performance.

[0535] Specific examples

[0536] The process will be explained as a concrete example when a user asks a robot, "What's the temperature today?"

[0537] 1. User says: "What time is it today?"

[0538] 2. The device's microphone captures the audio and sends the audio data to the server.

[0539] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[0540] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[0541] 5. The server uses a speech generation AI to generate a response text saying, "The current temperature is 25 degrees."

[0542] 6. The server uses a speech synthesis engine to convert the response text into voice data.

[0543] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[0544] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[0545] 9. The server receives the feedback and retrains the system.

[0546] This will enable natural and fluent dialogue between the robot and the user, resulting in a system that improves the user experience.

[0547] The processing flow will be explained below.

[0548] Specific processing steps of the program

[0549] Step 1:

[0550] The user speaks to the robot. At this time, the user's utterance is input as voice data, such as "What is the temperature today?"

[0551] Step 2:

[0552] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[0553] Step 3:

[0554] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[0555] Step 4:

[0556] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the user's intent: "I'm asking about the current temperature."

[0557] Step 5:

[0558] Based on the analysis results, the server calls a response generation module, which sends a request to an external API that provides temperature information. For example, the API returns the data "25 degrees."

[0559] Step 6:

[0560] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text. The generated response text is "The current temperature is 25 degrees."

[0561] Step 7:

[0562] The server inputs the generated response text into a text-to-speech engine (TTS), which converts the text data into speech data, generating speech that is easy for the user to understand.

[0563] Step 8:

[0564] The server sends the converted audio data to the device, which then plays it back through the device's speaker.

[0565] Step 9:

[0566] The user hears the robot's response. This time, the user receives the response, "The current temperature is 25 degrees."

[0567] Step 10:

[0568] If the user feels that the answer is not accurate, they can ask questions again or provide additional feedback. This feedback information is again collected by the device's microphone and sent to the server.

[0569] Step 11:

[0570] The server receives user feedback and adds it to the system's learning database, which allows the system to retrain and improve the accuracy of future responses.

[0571] This completes the process of an automated conversation system that utilizes voice generation AI.

[0572] Example 1

[0573] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] Conventional automated conversation systems have difficulty accurately understanding user intent and generating appropriate responses. Furthermore, they lack the ability to incorporate user feedback into learning to improve the system's accuracy, resulting in insufficient improvement in the user experience. Furthermore, the system's ability to effectively obtain external information and generate responses based on it is also limited. This leaves users unable to have satisfying conversations.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0576] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating response text using a generative AI model based on the analysis results, and means for converting the generated response text into voice data. This enables the server to generate appropriate and natural responses to user questions and improve the user experience. Furthermore, by including means for receiving feedback from the user and using it to learn and improve the system, the accuracy of the system can be improved. Furthermore, by including means for acquiring external information based on the analysis results, the server can provide more comprehensive responses.

[0577] The "means for collecting user speech as voice data" refers to a device or method for inputting the user's speech as a digital signal.

[0578] The "means for converting collected voice data into text data" refers to a technique or device that analyzes collected voice data and converts it into corresponding text data.

[0579] The "means for analyzing the converted text data and understanding the user's intent" refers to a technique or method for analyzing the text data using natural language processing technology and identifying the user's speech intent.

[0580] "Means for generating response text using a generative AI model based on analysis results" refers to techniques and methods that utilize a generative AI model to create appropriate response text based on analysis results that understand the user's intent.

[0581] The "means for converting the generated response text into voice data" refers to a technique or device that converts the generated response in text format into voice data using voice synthesis technology.

[0582] "Means for receiving user feedback and using it to learn and improve the system" refers to techniques and methods for collecting feedback information provided by users and incorporating it into the re-learning process using machine learning algorithms to improve the system and increase its accuracy.

[0583] "Means for acquiring external information based on the analysis results" refers to techniques or methods for acquiring the necessary external information from external sources such as databases or the Internet in accordance with the analyzed user's intentions.

[0584] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[0585] Voice input means

[0586] When a user speaks to the robot, their voice is collected by the device's microphone. The collected voice data is sent to the server in real time. Examples of voice input methods include a smartphone or a dedicated voice input device.

[0587] Converting audio data to text

[0588] The server that receives the voice data converts the collected voice data into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, a user's utterance, "What time is it today?", is generated as text data. This process includes feature extraction and phoneme recognition technology to improve the accuracy of the voice-to-text conversion.

[0589] Analysis of speech content

[0590] The converted text data is sent to the server's natural language processing (NLP) engine. NLP engines such as SpaCy and Google Cloud Natural Language API are used. This analyzes the text data and understands the user's intent. At this point, the system recognizes the intent to "ask about the current temperature."

[0591] Generate a response

[0592] Based on the analysis results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response text. For example, a response text such as "The current temperature is 25 degrees" is generated. An example of an input prompt for the generative AI model is as follows:

[0593] "The user asked 'What's the temperature today?' Generate a response."

[0594] Response text transcription

[0595] The generated response text is converted into voice data using a speech synthesis engine such as the Google Text-to-Speech API on the server. The speech synthesis engine has phonetic conversion technology and a speech waveform generation algorithm for converting text into speech waveforms.

[0596] Audio output

[0597] The generated voice data is sent from the server to the device. It is played back as voice from the device's speaker, and the user can hear the reply, "The current temperature is 25 degrees." The voice output means can be the robot's speaker or a smartphone speaker.

[0598] Gathering feedback and learning

[0599] If a user's response is incorrect or inappropriate, they can provide corrections or additional feedback. This feedback is sent to the server and added to the system's learning database. The server then uses this feedback to retrain the system to improve its performance. Machine learning frameworks such as TensorFlow are used for retraining.

[0600] Specific example explanation

[0601] As a concrete example, if a user asks a robot "What time is it today?", the process is as follows:

[0602] 1. User says: "What time is it today?"

[0603] 2. The device's microphone captures the audio and sends the audio data to the server.

[0604] 3. The server uses the Google Speech-to-Text API to convert the voice data into text: "What time is it today?"

[0605] 4. The server uses an NLP engine to parse the text and understand the intent: asking about the current temperature.

[0606] 5. The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[0607] 6. The server uses the Google Text-to-Speech API to convert the response text into audio data.

[0608] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[0609] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[0610] 9. The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[0611] This series of processes enables the user to have natural and fluent conversations with the robot, resulting in a system that improves the user experience.

[0612] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0613] Step 1:

[0614] User says: "What time is it today?"

[0615] Input: User's spoken utterance

[0616] Output: Digital audio data captured by the device's microphone

[0617] How it works: The user speaks to the robot, and their voice is captured through the device's microphone. At this stage, the voice is temporarily stored as digital data in a buffer.

[0618] Step 2:

[0619] The device's microphone captures the audio and sends the audio data to the server.

[0620] Input: Digital audio data

[0621] Output: Digital audio data sent to the server

[0622] Operation: The terminal transmits the captured audio data to the server in real time. The audio data is transmitted over the network using the TCP / IP protocol.

[0623] Step 3:

[0624] The server converts the audio data into text using the Google Speech-to-Text API.

[0625] Input: Digital audio data sent to the server

[0626] Output: Text data "What's the temperature today?"

[0627] How it works: The server passes the received audio data to the Google Speech-to-Text API, which converts the audio data into phonemes and generates the corresponding text data based on them.

[0628] Step 4:

[0629] The server uses an NLP engine to parse the text and understand the intent of asking about the current temperature.

[0630] Input: Text data "What's the temperature today?"

[0631] Output: Parsing the intent "Ask about the current temperature"

[0632] How it works: The text data is sent to the server's NLP engine (e.g., SpaCy or Google Cloud Natural Language API), which uses intent recognition algorithms to analyze the user's question intent.

[0633] Step 5:

[0634] The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[0635] Input: Parsed intent "Ask about the current temperature", prompt: "The user asked 'What's the temperature today?' Generate a response."

[0636] Output: Response text "The current temperature is 25 degrees."

[0637] How it works: The server provides the generative AI model with an input prompt and generates an appropriate response text based on the intent. The generative AI model creates this response based on a pre-trained dataset.

[0638] Step 6:

[0639] The server converts the response text into audio data using the Google Text-to-Speech API.

[0640] Input: Response text "The current temperature is 25 degrees"

[0641] Output: Audio data saying "The current temperature is 25 degrees"

[0642] How it works: The generated response text is sent to the server's Google Text-to-Speech API, where phonetic conversion from text to speech and generation of speech waveforms are performed, resulting in audio data.

[0643] Step 7:

[0644] The device plays the generated audio data over the speaker: "The current temperature is 25 degrees."

[0645] Input: Speech data saying "The current temperature is 25 degrees"

[0646] Output: Voice played through speaker: "The current temperature is 25 degrees."

[0647] Operation: Audio data is sent from the server to the device and output to the user as audio through the device's speaker.

[0648] Step 8:

[0649] The user listens to the response and may ask further questions or provide additional feedback.

[0650] Input: "The current temperature is 25 degrees"

[0651] Output: User feedback

[0652] How it works: The user listens to the reply and, if necessary, enters corrections or additional feedback into the terminal, which is then sent to the server.

[0653] Step 9:

[0654] The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[0655] Input: Feedback data received from users

[0656] Output: Improved model

[0657] How it works: Feedback data is stored on a server and the model is retrained using a machine learning framework such as TensorFlow, improving the accuracy of the system and the user experience.

[0658] (Application example 1)

[0659] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Conventional autonomous vehicles have difficulty in naturally interacting with users, requiring drivers and passengers to perform complex operations to set destinations, obtain current location information, and receive advice on safe driving. Furthermore, there has been a lack of easy ways to obtain external information in real time, such as weather information notifications, which has hindered the user experience of autonomous driving.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0662] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating a response text based on the analysis result, means for converting the generated response text into voice data, means for outputting the converted voice data to the user, and means for functioning as a voice assistant in an autonomous vehicle and interactively setting a destination, providing current location information, notifying weather information, and providing advice on safe driving. This enables multifunctional interaction through natural conversation with the user in an autonomous vehicle.

[0663] A "means for collecting user speech as audio data" is a device or method that captures the user's speech and stores it as digital audio data.

[0664] The "means for converting collected voice data into text data" refers to a process of converting voice data into corresponding character strings using voice recognition technology.

[0665] "Means for analyzing the converted text data and understanding the user's intent" refers to a function that uses natural language processing technology to analyze the text data and understand the user's intent and questions from its content.

[0666] The "means for generating a response text based on the analysis results" is a process for generating an appropriate response content in response to the analyzed user intent.

[0667] The "means for converting the generated response text into voice data" is a voice synthesis technology for generating natural and fluent voice from text.

[0668] The "means for outputting the converted voice data to the user" refers to a device or method for making the generated voice data audible to the user through a speaker or the like.

[0669] "A means that functions as a voice assistant within an autonomous vehicle and interactively provides destination setting, current location information, weather information, and advice on safe driving" refers to a system or software that can interact with the user via voice within the environment of the autonomous vehicle and respond to various information and instructions.

[0670] System Program Overview

[0671] A system for implementing this invention includes the following major components:

[0672] 1. Voice input means (terminal)

[0673] 2. Speech recognition means (server)

[0674] 3. Natural language processing means (server)

[0675] 4. Response Generation Method (Server)

[0676] 5. Speech synthesis means (server)

[0677] 6. Audio output means (terminal)

[0678] 7. Feedback Collection Method (Server)

[0679] Hardware and Software

[0680] The system uses a terminal equipped with a microphone and speaker to collect the user's speech in real time. The server uses the following software to process and calculate the data:

[0681] Speech recognition: Use the speech_recognition library to convert voice data into text data.

[0682] Natural language processing and response generation: Uses OpenAI's API to analyze user intent and generate appropriate response text.

[0683] Speech synthesis: Use the gtts library to convert response text into voice data.

[0684] Audio output: The generated audio data is played through the speaker using the playsound library.

[0685] Feedback collection: Receive feedback from users and use it to train the system.

[0686] Specific examples

[0687] User questions and how to handle them

[0688] For example, consider the case where a user says, "How much battery power is currently remaining in my vehicle?" The device's microphone captures the voice and sends the voice data to the server. The server then uses a speech recognition engine to convert this voice data into text. The server then analyzes this text with a natural language processing engine to understand the intent of the user's question. Based on the analysis results, a response generation engine generates a response text such as "The current battery power remaining is 80%." The generated response text is then converted into voice data using a speech synthesis engine and sent to the device. The device's speaker plays this voice data, conveying the response to the user.

[0689] Prompt Sentence Examples

[0690] Specific prompts are used to guide the generative AI model to generate appropriate responses. For example, here are some examples of prompts that can be used when a user asks:

[0691] Answer the following question: "How much battery power is left in my current vehicle?"

[0692] Based on this prompt, the system generates an appropriate response and outputs it as voice data.

[0693] Feedback collection and learning

[0694] If a user provides feedback on a response, that feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve the system's performance. For example, if a user provides feedback indicating dissatisfaction with a response, the system will adjust to provide better responses to similar questions in the future.

[0695] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0696] Step 1:

[0697] A means of collecting user utterances

[0698] When a user speaks in an autonomous vehicle, the device's microphone captures the voice. This voice data is collected in digital form and sent to a server. The input is the user's voice, and the output is digital voice data.

[0699] Step 2:

[0700] A means of converting audio data into text data

[0701] The server converts the received voice data into text data using a speech recognition engine (speech_recognition library). In this process, the voice waveform data is analyzed and the corresponding text is generated. The input is digital voice data, and the output is text data.

[0702] Step 3:

[0703] A means of analyzing text data and understanding user intent

[0704] The server analyzes the converted text data using a natural language processing engine (OpenAI API) to understand the user's intent. This analysis identifies what the user is looking for based on the content of the input text. The input is text data, and the output is the user's intent (for example, checking the remaining battery level).

[0705] Step 4:

[0706] A way to generate response text

[0707] Based on the analysis results, the server uses a response generation engine (OpenAI API) to generate an appropriate response text. The generative AI model creates a response text based on the prompt text. The input is the user's intention, and the output is the response text. An example of a specific prompt text is "Please answer the following question: 'How much battery is left in your current vehicle?'"

[0708] Step 5:

[0709] A means of converting response text into audio data

[0710] The server converts the generated response text into voice data using a speech synthesis engine (gtts library). Natural and fluent speech is generated from the text. The input is the response text, and the output is voice data.

[0711] Step 6:

[0712] A means for outputting audio data to the user

[0713] The server sends the generated voice data to the device and plays it from the device's speaker, allowing the user to hear the reply. The input is voice data, and the output is voice played from the speaker.

[0714] Step 7:

[0715] Gathering feedback and learning

[0716] Users can provide feedback on responses. For example, if a response is inappropriate, the user can offer corrections or suggestions for clarification. This feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve system performance. The input is the user's feedback, and the output is an updated learning database.

[0717] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0718] Detailed Description of the Invention

[0719] The present invention provides an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. This system not only collects and analyzes user utterances to generate responses, but also recognizes the user's emotions and responds based on those emotions. Specific embodiments of the system are described below.

[0720] The system mainly consists of the following components:

[0721] 1. Voice input means (terminal)

[0722] 2. Speech recognition means (server)

[0723] 3. Natural language processing means (server)

[0724] 4. Emotion Recognition Method (Server)

[0725] 5. Response Generation Method (Server)

[0726] 6. Speech synthesis means (server)

[0727] 7. Audio output means (terminal)

[0728] 8. Feedback Collection Method (Server)

[0729] A means of collecting user utterances

[0730] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[0731] Converting audio data to text

[0732] The server converts the collected voice data into text data using an automatic speech recognition (ASR) engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[0733] Analysis of speech content

[0734] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes that the user is asking about the current temperature.

[0735] Emotion recognition

[0736] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from their speech. For example, the utterance "What's the temperature today?" is recognized as "excited" or "calm."

[0737] Generate a response

[0738] Based on the analysis results and emotion recognition results, the server uses speech generation AI to generate an appropriate response text. For example, it might generate a response such as "The current temperature is 25 degrees." The response text is adjusted according to the user's emotions. If the user is excited, the response will be adjusted to sound calm.

[0739] Response text transcription

[0740] The server converts the generated response text into voice data using a text-to-speech engine (TTS). At this time, the tone and speed of the voice are adjusted based on the user's emotions. For example, if the user is calm, the response will be in a calm tone.

[0741] Audio output

[0742] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the response adjusted with the appropriate tone and speed.

[0743] Gathering feedback and learning

[0744] If the user's response is incorrect or inappropriate, they can make corrections or clarifications. This feedback is collected by the device's microphone and sent back to the server.

[0745] The server receives the feedback and adds it to the system's learning database, which allows the system to retrain itself and improve the accuracy of future responses.

[0746] Specific examples

[0747] Let us take a concrete example of the process when a user asks the robot in an excited voice, "What's the temperature today?"

[0748] 1. User says: "What's the temperature today?" (in an excited voice)

[0749] 2. The device's microphone captures the audio and sends the audio data to the server.

[0750] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[0751] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[0752] 5. The server uses an emotion recognition engine to recognize the emotion of the utterance as "excited."

[0753] 6. The server calls the response generation module based on the analysis results and emotion recognition results, and the speech generation AI generates the response text, "The current temperature is 25 degrees."

[0754] 7. The server converts the response text into voice data using a speech synthesis engine, responding in a calm tone to an excited user.

[0755] 8. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[0756] 9. The user listens to the response and possibly asks further questions or provides additional feedback.

[0757] 10. The server receives user feedback and retrains the system.

[0758] This will enable automated conversation systems that utilize voice generation AI and emotion recognition engines to achieve more natural and flexible dialogue, improving the user experience.

[0759] The processing flow will be explained below.

[0760] Detailed Description of the Invention

[0761] The specific processing steps of a conversation system incorporating an emotion recognition engine will be described below.

[0762] Step 1:

[0763] The user speaks to the robot, for example, saying, "What's the temperature today?"

[0764] Step 2:

[0765] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[0766] Step 3:

[0767] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[0768] Step 4:

[0769] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the intent that the user is asking about the current temperature.

[0770] Step 5:

[0771] At the same time, the server passes the voice data to an emotion recognition engine to analyze the user's emotions, which then recognizes the user's speech as "excited" or "calm."

[0772] Step 6:

[0773] Based on the analysis results (intent and emotion), the server calls the response generation module. This module sends a request to an external API to obtain temperature information. For example, the API returns the data "25 degrees."

[0774] Step 7:

[0775] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text, which is "The current temperature is 25 degrees."

[0776] Step 8:

[0777] The server inputs the response text into a text-to-speech engine (TTS) and converts the text data into voice data. During voice synthesis, the tone and speed of the voice are adjusted according to the user's emotions. For example, if the user is excited, the server will respond in a calm tone.

[0778] Step 9:

[0779] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[0780] Step 10:

[0781] If the user feels that the answer is not accurate, they can ask the question again or provide additional feedback, which is again collected by the device's microphone and sent to the server.

[0782] Step 11:

[0783] The server receives user feedback and adds it to the system's learning database, allowing the system to retrain based on the feedback and improve the accuracy of future responses.

[0784] This enables speech generation AI systems incorporating emotion recognition engines to enable more natural and emotionally appropriate dialogue, improving the user experience.

[0785] Example 2

[0786] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0787] Conventional automated conversation systems can collect user utterances, convert them into text data, and generate responses. However, they are unable to interpret the user's emotions and respond appropriately, resulting in a lack of naturalness in the dialogue and a poor user experience. Furthermore, it is difficult to utilize feedback to continuously improve the system and respond to diverse user needs.

[0788] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0789] In this invention, the server includes means for recognizing emotions from user utterances, means for generating response text based on the analysis results and emotion recognition results, and means for receiving feedback from the user and using it to learn and improve the system, thereby enabling more natural and effective dialogue that takes user emotions into consideration and allowing the system to learn and improve accuracy.

[0790] "User utterance" refers to a linguistic expression that a user verbally inputs to the system.

[0791] "Voice data" refers to the user's speech converted into a digital signal.

[0792] A "collection means" is a device or program that captures a user's voice data and stores it for processing.

[0793] "Text data" refers to character string information obtained by analyzing voice data.

[0794] "Means for converting" refers to devices or software for converting voice data into text data.

[0795] "Means of analysis" refers to machine learning models and programs that break down and analyze text data to understand its content and intent.

[0796] "User intent" refers to the information or request that a user wants to convey through speech.

[0797] "Means for recognizing emotions" refers to algorithms or programs that determine the emotions expressed by the user at that time from their speech.

[0798] "Analysis results" refers to the information obtained after interpreting the content and intent of text data.

[0799] The "emotion recognition result" is the classification result of emotions analyzed from the user's speech.

[0800] "Response text" is a character string representing the response content corresponding to the user's utterance.

[0801] "Generating means" refers to a device or program that creates a response text based on the analysis results and emotion recognition results.

[0802] "Means for converting into voice data" refers to the device or software that converts the response text back into voice form.

[0803] "Output means" refers to a device or program that plays the converted audio data on a speaker or the like so that the user can hear it.

[0804] "Feedback" refers to the evaluation or opinion that a user gives to the system's response.

[0805] "Means used to learn and improve the system" refers to machine learning algorithms and programs that incorporate user feedback to improve the system's performance and response accuracy.

[0806] This invention provides an automatic conversation system that utilizes a voice generation AI and an emotion recognition engine. Specifically, it is designed not only to collect user utterances, analyze them, and generate responses, but also to recognize the user's emotions and respond based on those emotions.

[0807] The system includes the following components:

[0808] 1. Voice input means (terminal)

[0809] 2. Speech recognition means (server)

[0810] 3. Natural language processing means (server)

[0811] 4. Emotion Recognition Method (Server)

[0812] 5. Response Generation Method (Server)

[0813] 6. Speech synthesis means (server)

[0814] 7. Audio output means (terminal)

[0815] 8. Feedback Collection Method (Server)

[0816] The device is equipped with a microphone that collects the user's voice. This microphone captures the voice spoken by the user and converts it into voice data. For example, if a user speaks to the robot, "What time is it today?", the voice is collected by the microphone.

[0817] The device sends the collected voice data to the server in real time. A secure protocol (e.g., HTTPS) is used for this transmission. The server then converts the received voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). For example, the voice "What's the temperature today?" is output as the text "What's the temperature today?"

[0818] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., GPT-3). During this process, it understands that the intent of the text is "asking about the current temperature." At the same time, the server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze the emotion from the user's speech. For example, the utterance "What's the temperature today?" is recognized as "excited."

[0819] Based on the analysis results and emotion recognition results, the server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text. For example, the response text generated might be, "The current temperature is 25 degrees." The tone of the response is adjusted according to the user's emotion. If the user is excited, the response will be set to a calm tone.

[0820] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Amazon Polly). The tone and speed of the voice are also adjusted based on the user's emotions. For example, if the response is made in a calm tone, the server will respond in a calm tone, saying, "The current temperature is 25 degrees."

[0821] The terminal receives the generated voice data and plays it back through the speaker. The user can hear the response, adjusted with the appropriate tone and speed. An example of a specific prompt sentence is the following dialogue when the user asks, "What time is it today?"

[0822] If the user adds or corrects a reply, that feedback is collected by the device's microphone and sent back to the server. The server receives the feedback, adds it to the system's learning database, and retrains it to improve the accuracy of future replies. This allows the system to continuously improve and provide more natural and accurate dialogue.

[0823] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0824] Step 1:

[0825] The user says, "What's the temperature today?"

[0826] Input: User's speech (voice data)

[0827] Output: Audio data captured by the device's microphone

[0828] Specific operation: The device's microphone captures the user's voice and obtains the voice data.

[0829] Step 2:

[0830] The device transmits the audio data to the server in real time.

[0831] Input: Audio data captured by the device's microphone

[0832] Output: Audio data sent to the server

[0833] What happens: The device sends the captured audio data to the server via a secure protocol (e.g., HTTPS).

[0834] Step 3:

[0835] The server converts the voice data into text data using a voice recognition engine.

[0836] Input: Transmitted audio data

[0837] Output: The converted text data (e.g. "What time is it today?")

[0838] Specific operation: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data.

[0839] Step 4:

[0840] The server uses a natural language processing engine to analyze the text data and understand the user's intent.

[0841] Input: Converted text data

[0842] Output: Analysis results indicating the user's intent (e.g., "I'm asking for the current temperature")

[0843] Specific operation: The server uses a natural language processing (NLP) engine (e.g., GPT-3) to analyze text data and understand the user's intent.

[0844] Step 5:

[0845] The server uses an emotion recognition engine to recognize emotions from the user's speech.

[0846] Input: Converted text data

[0847] Output: Recognition result indicating the user's emotion (e.g. excited)

[0848] Specific operation: The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze emotions from the user's speech.

[0849] Step 6:

[0850] The server generates a response text based on the analysis results and emotion recognition results.

[0851] Input: Analysis results and emotion recognition results that indicate the user's intentions

[0852] Output: Generated response text (e.g. "The current temperature is 25 degrees")

[0853] Specific operation: The server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text.

[0854] Step 7:

[0855] The server converts the generated response text into audio data.

[0856] Input: Generated response text

[0857] Output: Converted audio data

[0858] What happens: The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the response text into speech data, adjusting the tone and speed of the speech.

[0859] Step 8:

[0860] The server sends the audio data to the terminal.

[0861] Input: Converted audio data

[0862] Output: Audio data sent to the device

[0863] Specific operation: The server sends the audio data to the device using a secure protocol (e.g. HTTPS).

[0864] Step 9:

[0865] The audio data received by the device is output from the speaker.

[0866] Input: Transmitted audio data

[0867] Output: What the user hears (e.g., "The current temperature is 25 degrees")

[0868] What happens: The response is played through the device speaker with a controlled voice tone and speed.

[0869] Step 10:

[0870] The user provides feedback on the response.

[0871] Input: User feedback (voice data)

[0872] Output: Feedback audio data captured by the device's microphone

[0873] Specific operation: The device's microphone captures the user's feedback and obtains the audio data.

[0874] Step 11:

[0875] The terminal transmits the feedback voice data to the server.

[0876] Input: Captured feedback audio data

[0877] Output: Feedback audio data sent to the server

[0878] Specific operation: The device sends feedback audio data to the server via a secure protocol (e.g., HTTPS).

[0879] Step 12:

[0880] The server receives the feedback, adds it to the system's learning database, and retrains it.

[0881] Input: Submitted feedback audio data

[0882] Output: Updated training database and improved system

[0883] Specific operation: The server analyzes the feedback, adds it to the system's learning database, and performs re-training to improve response accuracy.

[0884] (Application example 2)

[0885] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0886] Conventional automated conversation systems can analyze user utterances and generate responses, but they face the challenge of generating appropriate responses that take the user's emotions into account. Furthermore, for users to safely and naturally converse in an autonomous vehicle while driving or traveling, responses that reflect the user's emotional state are required. This calls for a more advanced dialogue system.

[0887] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intention, means for recognizing emotions from the user's utterance, means for generating a response text based on the analysis result and the emotion recognition result, means for converting the generated response text into voice data, and means for outputting the converted voice data to the user. This enables an appropriate response according to the user's emotions.

[0888] "User" refers to a person who uses the system.

[0889] "Utterance" refers to words spoken by a user.

[0890] "Voice data" refers to data that is an electronic recording of a user's speech.

[0891] "Collection means" refers to a device or method for acquiring audio data.

[0892] "Text data" refers to data that has been converted from voice data into text information.

[0893] "Conversion means" refers to a device or method for converting audio data into text data.

[0894] "Analysis means" refers to a device or method that analyzes text data and understands the user's intent and content.

[0895] "Emotion recognition means" refers to a device or method for identifying emotions from a user's speech.

[0896] "Response generation means" refers to a device or method that generates an appropriate response based on the analysis results and emotion recognition results.

[0897] "Speech synthesis means" refers to a device or method that converts generated text responses into speech.

[0898] "Audio output means" refers to a device or method for reproducing audio data and allowing the user to listen to it.

[0899] "Feedback collection means" refers to a device or method for collecting corrections or supplemental information from users.

[0900] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[0901] This invention is an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. A specific embodiment for collecting user utterances in an autonomous vehicle and generating appropriate responses is described below.

[0902] System Overview

[0903] The system mainly consists of the following components:

[0904] 1. Voice input means (terminal)

[0905] 2. Speech recognition means (server)

[0906] 3. Natural language processing means (server)

[0907] 4. Emotion Recognition Method (Server)

[0908] 5. Response Generation Method (Server)

[0909] 6. Speech synthesis means (server)

[0910] 7. Audio output means (terminal)

[0911] 8. Feedback Collection Method (Server)

[0912] Voice input means

[0913] The device includes a microphone placed inside the autonomous vehicle to collect the user's voice, and the microphone is equipped with noise cancellation to ensure high-quality capture of the passenger's voice.

[0914] Voice recognition means

[0915] The server converts the collected voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API), thereby obtaining the user's speech as text data.

[0916] Natural language processing tools

[0917] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., Transformers Library). This analysis helps to understand the user's intent and the content of the question.

[0918] emotion recognition means

[0919] At the same time, the server uses an emotion recognition engine (e.g., Hugging Face emotion analysis model) to analyze the emotions from the user's speech. Emotion recognition identifies the user's state of mind based on the content of the text data.

[0920] Response generation method

[0921] The server uses a response generation AI (e.g., rinna / japanese-gpt2-medium) to generate appropriate response text based on the analysis and emotion recognition results. The generated response is tailored to the user's emotions.

[0922] Voice synthesis means

[0923] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Google Cloud Text-to-Speech API). During this conversion process, the voice tone and speed are adjusted according to the user's emotions.

[0924] Audio output means

[0925] The voice data is sent to the terminal and played back through the vehicle's speakers, allowing the user to receive a voice response.

[0926] Feedback collection methods

[0927] If a user's response is incorrect or inappropriate, they can make corrections or additions. This feedback is collected by the device's microphone and sent to the server, which receives the feedback and adds it to the system's learning database to improve the accuracy of the responses.

[0928] Specific examples

[0929] For example, if a user in a self-driving car asks "How long until the next service stop?" (in a worried voice), the system might:

[0930] 1. The device's microphone captures audio and sends the audio data to the server.

[0931] 2. The server uses a speech recognition engine to convert the voice data into text.

[0932] 3. The server analyzes the text using a natural language processing engine and understands the question.

[0933] 4. The server uses its emotion recognition engine to recognize the emotion of the utterance as "worry."

[0934] 5. The server uses a response generation AI to generate a response text that reads, "The next service area is about 30 minutes away. Please relax."

[0935] 6. When the server converts the response text into voice data using a speech synthesis engine, it responds in a gentle tone.

[0936] 7. The device plays the generated audio data from the speaker.

[0937] Prompt Sentence Examples

[0938] Input: "As a concierge AI for a self-driving vehicle, you are asked by a concerned passenger how long it will be to the next stop. Please create a response that will ease their concern."

[0939] Generate: "The next service area is about 30 minutes away. Relax."

[0940] This enables an automated conversation system that utilizes voice generation AI and an emotion recognition engine to provide flexible responses according to the user's emotional state, improving the user experience.

[0941] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0942] Step 1:

[0943] The device's microphone collects the user's speech and captures it as voice data. The input is the user's speech, and the output is voice data. This voice data is stored on the device as an electronic signal.

[0944] Step 2:

[0945] The device sends the collected voice data to the server. The input is the voice data stored on the device, and the output is the voice data transferred to the server.

[0946] Step 3:

[0947] The server uses a speech recognition engine to convert the voice data into text data. The input is voice data and the output is text data. The speech recognition engine (e.g., Google Speech-to-Text API) performs this conversion.

[0948] Step 4:

[0949] The server uses a natural language processing engine to analyze the text data and understand the user's intent. The input is text data, and the output is an analysis result that indicates the user's intent. The natural language processing engine (e.g., Transformers Library) performs this analysis.

[0950] Step 5:

[0951] The server uses an emotion recognition engine to recognize the user's emotions from the text data. The input is the text data, and the output is the emotion recognition result. The emotion recognition engine (e.g., Hugging Face emotion analysis model) performs this analysis.

[0952] Step 6:

[0953] The server generates a response text using a response generation AI based on the analysis results and emotion recognition results. The input is the analysis results and emotion recognition results, and the output is the response text. This generation is performed by the response generation AI (e.g., rinna / japanese-gpt2-medium).

[0954] Step 7:

[0955] The server uses a speech synthesis engine to convert the generated response text into audio data. The input is the response text and the output is audio data. The speech synthesis engine (e.g., Google Cloud Text-to-Speech API) performs this conversion.

[0956] Step 8:

[0957] The server sends the voice data to the terminal. The input is the voice data generated by the server, and the output is the voice data transferred to the terminal.

[0958] Step 9:

[0959] The device plays the received audio data from the speaker. The input is the audio data stored on the device, and the output is the audio played from the speaker.

[0960] Step 10:

[0961] The user hears the reply and provides feedback on the content. The input is the user's recognition and the output is the feedback speech.

[0962] Step 11:

[0963] The terminal captures the feedback voice from the user and sends it to the server. The input is the feedback voice, and the output is the feedback voice transferred to the server.

[0964] Step 12:

[0965] The server analyzes the feedback speech and uses it to improve the system. The input is the feedback speech, and the output is the system's learning and improvement results. This allows the system to generate more accurate responses.

[0966] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0967] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0968] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0969] [Third embodiment]

[0970] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0971] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0972] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0973] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0974] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0975] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0976] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0977] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0978] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0979] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0980] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0981] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0982] Detailed Description of the Invention

[0983] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[0984] The system mainly consists of the following components:

[0985] 1. Voice input means (terminal)

[0986] 2. Speech recognition means (server)

[0987] 3. Natural language processing means (server)

[0988] 4. Response Generation Method (Server)

[0989] 5. Speech synthesis means (server)

[0990] 6. Audio output means (terminal)

[0991] 7. Feedback Collection Method (Server)

[0992] A means of collecting user utterances

[0993] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[0994] Converting audio data to text

[0995] The server converts the collected voice data into text data using a voice recognition engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[0996] Analysis of speech content

[0997] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes the intent as "ask about the current temperature."

[0998] Generate a response

[0999] Based on the analysis results, the server uses speech generation AI to generate an appropriate response, such as "The current temperature is 25 degrees."

[1000] Response text transcription

[1001] The server converts the generated response text into voice data using a speech synthesis engine, which generates a voice that the user perceives as natural speech.

[1002] Audio output

[1003] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[1004] Gathering feedback and learning

[1005] If the user's response is inaccurate or inappropriate, they can make corrections or additions. This feedback is sent to the server and added to the system's learning database. The server uses the feedback to retrain the system to improve its performance.

[1006] Specific examples

[1007] The process will be explained as a concrete example when a user asks a robot, "What's the temperature today?"

[1008] 1. User says: "What time is it today?"

[1009] 2. The device's microphone captures the audio and sends the audio data to the server.

[1010] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[1011] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[1012] 5. The server uses a speech generation AI to generate a response text saying, "The current temperature is 25 degrees."

[1013] 6. The server uses a speech synthesis engine to convert the response text into voice data.

[1014] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[1015] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[1016] 9. The server receives the feedback and retrains the system.

[1017] This will enable natural and fluent dialogue between the robot and the user, resulting in a system that improves the user experience.

[1018] The processing flow will be explained below.

[1019] Specific processing steps of the program

[1020] Step 1:

[1021] The user speaks to the robot. At this time, the user's utterance is input as voice data, such as "What is the temperature today?"

[1022] Step 2:

[1023] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[1024] Step 3:

[1025] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[1026] Step 4:

[1027] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the user's intent: "I'm asking about the current temperature."

[1028] Step 5:

[1029] Based on the analysis results, the server calls a response generation module, which sends a request to an external API that provides temperature information. For example, the API returns the data "25 degrees."

[1030] Step 6:

[1031] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text. The generated response text is "The current temperature is 25 degrees."

[1032] Step 7:

[1033] The server inputs the generated response text into a text-to-speech engine (TTS), which converts the text data into speech data, generating speech that is easy for the user to understand.

[1034] Step 8:

[1035] The server sends the converted audio data to the device, which then plays it back through the device's speaker.

[1036] Step 9:

[1037] The user hears the robot's response. This time, the user receives the response, "The current temperature is 25 degrees."

[1038] Step 10:

[1039] If the user feels that the answer is not accurate, they can ask questions again or provide additional feedback. This feedback information is again collected by the device's microphone and sent to the server.

[1040] Step 11:

[1041] The server receives user feedback and adds it to the system's learning database, which allows the system to retrain and improve the accuracy of future responses.

[1042] This completes the process of an automated conversation system that utilizes voice generation AI.

[1043] Example 1

[1044] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1045] Conventional automated conversation systems have difficulty accurately understanding user intent and generating appropriate responses. Furthermore, they lack the ability to incorporate user feedback into learning to improve the system's accuracy, resulting in insufficient improvement in the user experience. Furthermore, the system's ability to effectively obtain external information and generate responses based on it is also limited. This leaves users unable to have satisfying conversations.

[1046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1047] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating response text using a generative AI model based on the analysis results, and means for converting the generated response text into voice data. This enables the server to generate appropriate and natural responses to user questions and improve the user experience. Furthermore, by including means for receiving feedback from the user and using it to learn and improve the system, the accuracy of the system can be improved. Furthermore, by including means for acquiring external information based on the analysis results, the server can provide more comprehensive responses.

[1048] The "means for collecting user speech as voice data" refers to a device or method for inputting the user's speech as a digital signal.

[1049] The "means for converting collected voice data into text data" refers to a technique or device that analyzes collected voice data and converts it into corresponding text data.

[1050] The "means for analyzing the converted text data and understanding the user's intent" refers to a technique or method for analyzing the text data using natural language processing technology and identifying the user's speech intent.

[1051] "Means for generating response text using a generative AI model based on analysis results" refers to techniques and methods that utilize a generative AI model to create appropriate response text based on analysis results that understand the user's intent.

[1052] The "means for converting the generated response text into voice data" refers to a technique or device that converts the generated response in text format into voice data using voice synthesis technology.

[1053] "Means for receiving user feedback and using it to learn and improve the system" refers to techniques and methods for collecting feedback information provided by users and incorporating it into the re-learning process using machine learning algorithms to improve the system and increase its accuracy.

[1054] "Means for acquiring external information based on the analysis results" refers to techniques or methods for acquiring the necessary external information from external sources such as databases or the Internet in accordance with the analyzed user's intentions.

[1055] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[1056] Voice input means

[1057] When a user speaks to the robot, their voice is collected by the device's microphone. The collected voice data is sent to the server in real time. Examples of voice input methods include a smartphone or a dedicated voice input device.

[1058] Converting audio data to text

[1059] The server that receives the voice data converts the collected voice data into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, a user's utterance, "What time is it today?", is generated as text data. This process includes feature extraction and phoneme recognition technology to improve the accuracy of the voice-to-text conversion.

[1060] Analysis of speech content

[1061] The converted text data is sent to the server's natural language processing (NLP) engine. NLP engines such as SpaCy and Google Cloud Natural Language API are used. This analyzes the text data and understands the user's intent. At this point, the system recognizes the intent to "ask about the current temperature."

[1062] Generate a response

[1063] Based on the analysis results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response text. For example, a response text such as "The current temperature is 25 degrees" is generated. An example of an input prompt for the generative AI model is as follows:

[1064] "The user asked 'What's the temperature today?' Generate a response."

[1065] Response text transcription

[1066] The generated response text is converted into voice data using a speech synthesis engine such as the Google Text-to-Speech API on the server. The speech synthesis engine has phonetic conversion technology and a speech waveform generation algorithm for converting text into speech waveforms.

[1067] Audio output

[1068] The generated voice data is sent from the server to the device. It is played back as voice from the device's speaker, and the user can hear the reply, "The current temperature is 25 degrees." The voice output means can be the robot's speaker or a smartphone speaker.

[1069] Gathering feedback and learning

[1070] If a user's response is incorrect or inappropriate, they can provide corrections or additional feedback. This feedback is sent to the server and added to the system's learning database. The server then uses this feedback to retrain the system to improve its performance. Machine learning frameworks such as TensorFlow are used for retraining.

[1071] Specific example explanation

[1072] As a concrete example, if a user asks a robot "What time is it today?", the process is as follows:

[1073] 1. User says: "What time is it today?"

[1074] 2. The device's microphone captures the audio and sends the audio data to the server.

[1075] 3. The server uses the Google Speech-to-Text API to convert the voice data into text: "What time is it today?"

[1076] 4. The server uses an NLP engine to parse the text and understand the intent: asking about the current temperature.

[1077] 5. The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[1078] 6. The server uses the Google Text-to-Speech API to convert the response text into audio data.

[1079] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[1080] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[1081] 9. The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[1082] This series of processes enables the user to have natural and fluent conversations with the robot, resulting in a system that improves the user experience.

[1083] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1084] Step 1:

[1085] User says: "What time is it today?"

[1086] Input: User's spoken utterance

[1087] Output: Digital audio data captured by the device's microphone

[1088] How it works: The user speaks to the robot, and their voice is captured through the device's microphone. At this stage, the voice is temporarily stored as digital data in a buffer.

[1089] Step 2:

[1090] The device's microphone captures the audio and sends the audio data to the server.

[1091] Input: Digital audio data

[1092] Output: Digital audio data sent to the server

[1093] Operation: The terminal transmits the captured audio data to the server in real time. The audio data is transmitted over the network using the TCP / IP protocol.

[1094] Step 3:

[1095] The server converts the audio data into text using the Google Speech-to-Text API.

[1096] Input: Digital audio data sent to the server

[1097] Output: Text data "What's the temperature today?"

[1098] How it works: The server passes the received audio data to the Google Speech-to-Text API, which converts the audio data into phonemes and generates the corresponding text data based on them.

[1099] Step 4:

[1100] The server uses an NLP engine to parse the text and understand the intent of asking about the current temperature.

[1101] Input: Text data "What's the temperature today?"

[1102] Output: Parsing the intent "Ask about the current temperature"

[1103] How it works: The text data is sent to the server's NLP engine (e.g., SpaCy or Google Cloud Natural Language API), which uses intent recognition algorithms to analyze the user's question intent.

[1104] Step 5:

[1105] The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[1106] Input: Parsed intent "Ask about the current temperature", prompt: "The user asked 'What's the temperature today?' Generate a response."

[1107] Output: Response text "The current temperature is 25 degrees."

[1108] How it works: The server provides the generative AI model with an input prompt and generates an appropriate response text based on the intent. The generative AI model creates this response based on a pre-trained dataset.

[1109] Step 6:

[1110] The server converts the response text into audio data using the Google Text-to-Speech API.

[1111] Input: Response text "The current temperature is 25 degrees"

[1112] Output: Audio data saying "The current temperature is 25 degrees"

[1113] How it works: The generated response text is sent to the server's Google Text-to-Speech API, where phonetic conversion from text to speech and generation of speech waveforms are performed, resulting in audio data.

[1114] Step 7:

[1115] The device plays the generated audio data over the speaker: "The current temperature is 25 degrees."

[1116] Input: Speech data saying "The current temperature is 25 degrees"

[1117] Output: Voice played through speaker: "The current temperature is 25 degrees."

[1118] Operation: Audio data is sent from the server to the device and output to the user as audio through the device's speaker.

[1119] Step 8:

[1120] The user listens to the response and may ask further questions or provide additional feedback.

[1121] Input: "The current temperature is 25 degrees"

[1122] Output: User feedback

[1123] How it works: The user listens to the reply and, if necessary, enters corrections or additional feedback into the terminal, which is then sent to the server.

[1124] Step 9:

[1125] The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[1126] Input: Feedback data received from users

[1127] Output: Improved model

[1128] How it works: Feedback data is stored on a server and the model is retrained using a machine learning framework such as TensorFlow, improving the accuracy of the system and the user experience.

[1129] (Application example 1)

[1130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1131] Conventional autonomous vehicles have difficulty in naturally interacting with users, requiring drivers and passengers to perform complex operations to set destinations, obtain current location information, and receive advice on safe driving. Furthermore, there has been a lack of easy ways to obtain external information in real time, such as weather information notifications, which has hindered the user experience of autonomous driving.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1133] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating a response text based on the analysis result, means for converting the generated response text into voice data, means for outputting the converted voice data to the user, and means for functioning as a voice assistant in an autonomous vehicle and interactively setting a destination, providing current location information, notifying weather information, and providing advice on safe driving. This enables multifunctional interaction through natural conversation with the user in an autonomous vehicle.

[1134] A "means for collecting user speech as audio data" is a device or method that captures the user's speech and stores it as digital audio data.

[1135] The "means for converting collected voice data into text data" refers to a process of converting voice data into corresponding character strings using voice recognition technology.

[1136] "Means for analyzing the converted text data and understanding the user's intent" refers to a function that uses natural language processing technology to analyze the text data and understand the user's intent and questions from its content.

[1137] The "means for generating a response text based on the analysis results" is a process for generating an appropriate response content in response to the analyzed user intent.

[1138] The "means for converting the generated response text into voice data" is a voice synthesis technology for generating natural and fluent voice from text.

[1139] The "means for outputting the converted voice data to the user" refers to a device or method for making the generated voice data audible to the user through a speaker or the like.

[1140] "A means that functions as a voice assistant within an autonomous vehicle and interactively provides destination setting, current location information, weather information, and advice on safe driving" refers to a system or software that can interact with the user via voice within the environment of the autonomous vehicle and respond to various information and instructions.

[1141] System Program Overview

[1142] A system for implementing this invention includes the following major components:

[1143] 1. Voice input means (terminal)

[1144] 2. Speech recognition means (server)

[1145] 3. Natural language processing means (server)

[1146] 4. Response Generation Method (Server)

[1147] 5. Speech synthesis means (server)

[1148] 6. Audio output means (terminal)

[1149] 7. Feedback Collection Method (Server)

[1150] Hardware and Software

[1151] The system uses a terminal equipped with a microphone and speaker to collect the user's speech in real time. The server uses the following software to process and calculate the data:

[1152] Speech recognition: Use the speech_recognition library to convert voice data into text data.

[1153] Natural language processing and response generation: Uses OpenAI's API to analyze user intent and generate appropriate response text.

[1154] Speech synthesis: Use the gtts library to convert response text into voice data.

[1155] Audio output: The generated audio data is played through the speaker using the playsound library.

[1156] Feedback collection: Receive feedback from users and use it to train the system.

[1157] Specific examples

[1158] User questions and how to handle them

[1159] For example, consider the case where a user says, "How much battery power is currently remaining in my vehicle?" The device's microphone captures the voice and sends the voice data to the server. The server then uses a speech recognition engine to convert this voice data into text. The server then analyzes this text with a natural language processing engine to understand the intent of the user's question. Based on the analysis results, a response generation engine generates a response text such as "The current battery power remaining is 80%." The generated response text is then converted into voice data using a speech synthesis engine and sent to the device. The device's speaker plays this voice data, conveying the response to the user.

[1160] Prompt Sentence Examples

[1161] Specific prompts are used to guide the generative AI model to generate appropriate responses. For example, here are some examples of prompts that can be used when a user asks:

[1162] Answer the following question: "How much battery power is left in my current vehicle?"

[1163] Based on this prompt, the system generates an appropriate response and outputs it as voice data.

[1164] Feedback collection and learning

[1165] If a user provides feedback on a response, that feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve the system's performance. For example, if a user provides feedback indicating dissatisfaction with a response, the system will adjust to provide better responses to similar questions in the future.

[1166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1167] Step 1:

[1168] A means of collecting user utterances

[1169] When a user speaks in an autonomous vehicle, the device's microphone captures the voice. This voice data is collected in digital form and sent to a server. The input is the user's voice, and the output is digital voice data.

[1170] Step 2:

[1171] A means of converting audio data into text data

[1172] The server converts the received voice data into text data using a speech recognition engine (speech_recognition library). In this process, the voice waveform data is analyzed and the corresponding text is generated. The input is digital voice data, and the output is text data.

[1173] Step 3:

[1174] A means of analyzing text data and understanding user intent

[1175] The server analyzes the converted text data using a natural language processing engine (OpenAI API) to understand the user's intent. This analysis identifies what the user is looking for based on the content of the input text. The input is text data, and the output is the user's intent (for example, checking the remaining battery level).

[1176] Step 4:

[1177] A way to generate response text

[1178] Based on the analysis results, the server uses a response generation engine (OpenAI API) to generate an appropriate response text. The generative AI model creates a response text based on the prompt text. The input is the user's intention, and the output is the response text. An example of a specific prompt text is "Please answer the following question: 'How much battery is left in your current vehicle?'"

[1179] Step 5:

[1180] A means of converting response text into audio data

[1181] The server converts the generated response text into voice data using a speech synthesis engine (gtts library). Natural and fluent speech is generated from the text. The input is the response text, and the output is voice data.

[1182] Step 6:

[1183] A means for outputting audio data to the user

[1184] The server sends the generated voice data to the device and plays it from the device's speaker, allowing the user to hear the reply. The input is voice data, and the output is voice played from the speaker.

[1185] Step 7:

[1186] Gathering feedback and learning

[1187] Users can provide feedback on responses. For example, if a response is inappropriate, the user can offer corrections or suggestions for clarification. This feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve system performance. The input is the user's feedback, and the output is an updated learning database.

[1188] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1189] Detailed Description of the Invention

[1190] The present invention provides an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. This system not only collects and analyzes user utterances to generate responses, but also recognizes the user's emotions and responds based on those emotions. Specific embodiments of the system are described below.

[1191] The system mainly consists of the following components:

[1192] 1. Voice input means (terminal)

[1193] 2. Speech recognition means (server)

[1194] 3. Natural language processing means (server)

[1195] 4. Emotion Recognition Method (Server)

[1196] 5. Response Generation Method (Server)

[1197] 6. Speech synthesis means (server)

[1198] 7. Audio output means (terminal)

[1199] 8. Feedback Collection Method (Server)

[1200] A means of collecting user utterances

[1201] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[1202] Converting audio data to text

[1203] The server converts the collected voice data into text data using an automatic speech recognition (ASR) engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[1204] Analysis of speech content

[1205] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes that the user is asking about the current temperature.

[1206] Emotion recognition

[1207] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from their speech. For example, the utterance "What's the temperature today?" is recognized as "excited" or "calm."

[1208] Generate a response

[1209] Based on the analysis results and emotion recognition results, the server uses speech generation AI to generate an appropriate response text. For example, it might generate a response such as "The current temperature is 25 degrees." The response text is adjusted according to the user's emotions. If the user is excited, the response will be adjusted to sound calm.

[1210] Response text transcription

[1211] The server converts the generated response text into voice data using a text-to-speech engine (TTS). At this time, the tone and speed of the voice are adjusted based on the user's emotions. For example, if the user is calm, the response will be in a calm tone.

[1212] Audio output

[1213] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the response adjusted with the appropriate tone and speed.

[1214] Gathering feedback and learning

[1215] If the user's response is incorrect or inappropriate, they can make corrections or clarifications. This feedback is collected by the device's microphone and sent back to the server.

[1216] The server receives the feedback and adds it to the system's learning database, which allows the system to retrain itself and improve the accuracy of future responses.

[1217] Specific examples

[1218] Let us take a concrete example of the process when a user asks the robot in an excited voice, "What's the temperature today?"

[1219] 1. User says: "What's the temperature today?" (in an excited voice)

[1220] 2. The device's microphone captures the audio and sends the audio data to the server.

[1221] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[1222] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[1223] 5. The server uses an emotion recognition engine to recognize the emotion of the utterance as "excited."

[1224] 6. The server calls the response generation module based on the analysis results and emotion recognition results, and the speech generation AI generates the response text, "The current temperature is 25 degrees."

[1225] 7. The server converts the response text into voice data using a speech synthesis engine, responding in a calm tone to an excited user.

[1226] 8. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[1227] 9. The user listens to the response and possibly asks further questions or provides additional feedback.

[1228] 10. The server receives user feedback and retrains the system.

[1229] This will enable automated conversation systems that utilize voice generation AI and emotion recognition engines to achieve more natural and flexible dialogue, improving the user experience.

[1230] The processing flow will be explained below.

[1231] Detailed Description of the Invention

[1232] The specific processing steps of a conversation system incorporating an emotion recognition engine will be described below.

[1233] Step 1:

[1234] The user speaks to the robot, for example, saying, "What's the temperature today?"

[1235] Step 2:

[1236] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[1237] Step 3:

[1238] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[1239] Step 4:

[1240] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the intent that the user is asking about the current temperature.

[1241] Step 5:

[1242] At the same time, the server passes the voice data to an emotion recognition engine to analyze the user's emotions, which then recognizes the user's speech as "excited" or "calm."

[1243] Step 6:

[1244] Based on the analysis results (intent and emotion), the server calls the response generation module. This module sends a request to an external API to obtain temperature information. For example, the API returns the data "25 degrees."

[1245] Step 7:

[1246] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text, which is "The current temperature is 25 degrees."

[1247] Step 8:

[1248] The server inputs the response text into a text-to-speech engine (TTS) and converts the text data into voice data. During voice synthesis, the tone and speed of the voice are adjusted according to the user's emotions. For example, if the user is excited, the server will respond in a calm tone.

[1249] Step 9:

[1250] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[1251] Step 10:

[1252] If the user feels that the answer is not accurate, they can ask the question again or provide additional feedback, which is again collected by the device's microphone and sent to the server.

[1253] Step 11:

[1254] The server receives user feedback and adds it to the system's learning database, allowing the system to retrain based on the feedback and improve the accuracy of future responses.

[1255] This enables speech generation AI systems incorporating emotion recognition engines to enable more natural and emotionally appropriate dialogue, improving the user experience.

[1256] Example 2

[1257] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1258] Conventional automated conversation systems can collect user utterances, convert them into text data, and generate responses. However, they are unable to interpret the user's emotions and respond appropriately, resulting in a lack of naturalness in the dialogue and a poor user experience. Furthermore, it is difficult to utilize feedback to continuously improve the system and respond to diverse user needs.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1260] In this invention, the server includes means for recognizing emotions from user utterances, means for generating response text based on the analysis results and emotion recognition results, and means for receiving feedback from the user and using it to learn and improve the system, thereby enabling more natural and effective dialogue that takes user emotions into consideration and allowing the system to learn and improve accuracy.

[1261] "User utterance" refers to a linguistic expression that a user verbally inputs to the system.

[1262] "Voice data" refers to the user's speech converted into a digital signal.

[1263] A "collection means" is a device or program that captures a user's voice data and stores it for processing.

[1264] "Text data" refers to character string information obtained by analyzing voice data.

[1265] "Means for converting" refers to devices or software for converting voice data into text data.

[1266] "Means of analysis" refers to machine learning models and programs that break down and analyze text data to understand its content and intent.

[1267] "User intent" refers to the information or request that a user wants to convey through speech.

[1268] "Means for recognizing emotions" refers to algorithms or programs that determine the emotions expressed by the user at that time from their speech.

[1269] "Analysis results" refers to the information obtained after interpreting the content and intent of text data.

[1270] The "emotion recognition result" is the classification result of emotions analyzed from the user's speech.

[1271] "Response text" is a character string representing the response content corresponding to the user's utterance.

[1272] "Generating means" refers to a device or program that creates a response text based on the analysis results and emotion recognition results.

[1273] "Means for converting into voice data" refers to the device or software that converts the response text back into voice form.

[1274] "Output means" refers to a device or program that plays the converted audio data on a speaker or the like so that the user can hear it.

[1275] "Feedback" refers to the evaluation or opinion that a user gives to the system's response.

[1276] "Means used to learn and improve the system" refers to machine learning algorithms and programs that incorporate user feedback to improve the system's performance and response accuracy.

[1277] This invention provides an automatic conversation system that utilizes a voice generation AI and an emotion recognition engine. Specifically, it is designed not only to collect user utterances, analyze them, and generate responses, but also to recognize the user's emotions and respond based on those emotions.

[1278] The system includes the following components:

[1279] 1. Voice input means (terminal)

[1280] 2. Speech recognition means (server)

[1281] 3. Natural language processing means (server)

[1282] 4. Emotion Recognition Method (Server)

[1283] 5. Response Generation Method (Server)

[1284] 6. Speech synthesis means (server)

[1285] 7. Audio output means (terminal)

[1286] 8. Feedback Collection Method (Server)

[1287] The device is equipped with a microphone that collects the user's voice. This microphone captures the voice spoken by the user and converts it into voice data. For example, if a user speaks to the robot, "What time is it today?", the voice is collected by the microphone.

[1288] The device sends the collected voice data to the server in real time. A secure protocol (e.g., HTTPS) is used for this transmission. The server then converts the received voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). For example, the voice "What's the temperature today?" is output as the text "What's the temperature today?"

[1289] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., GPT-3). During this process, it understands that the intent of the text is "asking about the current temperature." At the same time, the server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze the emotion from the user's speech. For example, the utterance "What's the temperature today?" is recognized as "excited."

[1290] Based on the analysis results and emotion recognition results, the server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text. For example, the response text generated might be, "The current temperature is 25 degrees." The tone of the response is adjusted according to the user's emotion. If the user is excited, the response will be set to a calm tone.

[1291] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Amazon Polly). The tone and speed of the voice are also adjusted based on the user's emotions. For example, if the response is made in a calm tone, the server will respond in a calm tone, saying, "The current temperature is 25 degrees."

[1292] The terminal receives the generated voice data and plays it back through the speaker. The user can hear the response, adjusted with the appropriate tone and speed. An example of a specific prompt sentence is the following dialogue when the user asks, "What time is it today?"

[1293] If the user adds or corrects a reply, that feedback is collected by the device's microphone and sent back to the server. The server receives the feedback, adds it to the system's learning database, and retrains it to improve the accuracy of future replies. This allows the system to continuously improve and provide more natural and accurate dialogue.

[1294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1295] Step 1:

[1296] The user says, "What's the temperature today?"

[1297] Input: User's speech (voice data)

[1298] Output: Audio data captured by the device's microphone

[1299] Specific operation: The device's microphone captures the user's voice and obtains the voice data.

[1300] Step 2:

[1301] The device transmits the audio data to the server in real time.

[1302] Input: Audio data captured by the device's microphone

[1303] Output: Audio data sent to the server

[1304] What happens: The device sends the captured audio data to the server via a secure protocol (e.g., HTTPS).

[1305] Step 3:

[1306] The server converts the voice data into text data using a voice recognition engine.

[1307] Input: Transmitted audio data

[1308] Output: The converted text data (e.g. "What time is it today?")

[1309] Specific operation: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data.

[1310] Step 4:

[1311] The server uses a natural language processing engine to analyze the text data and understand the user's intent.

[1312] Input: Converted text data

[1313] Output: Analysis results indicating the user's intent (e.g., "I'm asking for the current temperature")

[1314] Specific operation: The server uses a natural language processing (NLP) engine (e.g., GPT-3) to analyze text data and understand the user's intent.

[1315] Step 5:

[1316] The server uses an emotion recognition engine to recognize emotions from the user's speech.

[1317] Input: Converted text data

[1318] Output: Recognition result indicating the user's emotion (e.g. excited)

[1319] Specific operation: The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze emotions from the user's speech.

[1320] Step 6:

[1321] The server generates a response text based on the analysis results and emotion recognition results.

[1322] Input: Analysis results and emotion recognition results that indicate the user's intentions

[1323] Output: Generated response text (e.g. "The current temperature is 25 degrees")

[1324] Specific operation: The server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text.

[1325] Step 7:

[1326] The server converts the generated response text into audio data.

[1327] Input: Generated response text

[1328] Output: Converted audio data

[1329] What happens: The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the response text into speech data, adjusting the tone and speed of the speech.

[1330] Step 8:

[1331] The server sends the audio data to the terminal.

[1332] Input: Converted audio data

[1333] Output: Audio data sent to the device

[1334] Specific operation: The server sends the audio data to the device using a secure protocol (e.g. HTTPS).

[1335] Step 9:

[1336] The audio data received by the device is output from the speaker.

[1337] Input: Transmitted audio data

[1338] Output: What the user hears (e.g., "The current temperature is 25 degrees")

[1339] What happens: The response is played through the device speaker with a controlled voice tone and speed.

[1340] Step 10:

[1341] The user provides feedback on the response.

[1342] Input: User feedback (voice data)

[1343] Output: Feedback audio data captured by the device's microphone

[1344] Specific operation: The device's microphone captures the user's feedback and obtains the audio data.

[1345] Step 11:

[1346] The terminal transmits the feedback voice data to the server.

[1347] Input: Captured feedback audio data

[1348] Output: Feedback audio data sent to the server

[1349] Specific operation: The device sends feedback audio data to the server via a secure protocol (e.g., HTTPS).

[1350] Step 12:

[1351] The server receives the feedback, adds it to the system's learning database, and retrains it.

[1352] Input: Submitted feedback audio data

[1353] Output: Updated training database and improved system

[1354] Specific operation: The server analyzes the feedback, adds it to the system's learning database, and performs re-training to improve response accuracy.

[1355] (Application example 2)

[1356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1357] Conventional automated conversation systems can analyze user utterances and generate responses, but they face the challenge of generating appropriate responses that take the user's emotions into account. Furthermore, for users to safely and naturally converse in an autonomous vehicle while driving or traveling, responses that reflect the user's emotional state are required. This calls for a more advanced dialogue system.

[1358] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intention, means for recognizing emotions from the user's utterance, means for generating a response text based on the analysis result and the emotion recognition result, means for converting the generated response text into voice data, and means for outputting the converted voice data to the user. This enables an appropriate response according to the user's emotions.

[1359] "User" refers to a person who uses the system.

[1360] "Utterance" refers to words spoken by a user.

[1361] "Voice data" refers to data that is an electronic recording of a user's speech.

[1362] "Collection means" refers to a device or method for acquiring audio data.

[1363] "Text data" refers to data that has been converted from voice data into text information.

[1364] "Conversion means" refers to a device or method for converting audio data into text data.

[1365] "Analysis means" refers to a device or method that analyzes text data and understands the user's intent and content.

[1366] "Emotion recognition means" refers to a device or method for identifying emotions from a user's speech.

[1367] "Response generation means" refers to a device or method that generates an appropriate response based on the analysis results and emotion recognition results.

[1368] "Speech synthesis means" refers to a device or method that converts generated text responses into speech.

[1369] "Audio output means" refers to a device or method for reproducing audio data and allowing the user to listen to it.

[1370] "Feedback collection means" refers to a device or method for collecting corrections or supplemental information from users.

[1371] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[1372] This invention is an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. A specific embodiment for collecting user utterances in an autonomous vehicle and generating appropriate responses is described below.

[1373] System Overview

[1374] The system mainly consists of the following components:

[1375] 1. Voice input means (terminal)

[1376] 2. Speech recognition means (server)

[1377] 3. Natural language processing means (server)

[1378] 4. Emotion Recognition Method (Server)

[1379] 5. Response Generation Method (Server)

[1380] 6. Speech synthesis means (server)

[1381] 7. Audio output means (terminal)

[1382] 8. Feedback Collection Method (Server)

[1383] Voice input means

[1384] The device includes a microphone placed inside the autonomous vehicle to collect the user's voice, and the microphone is equipped with noise cancellation to ensure high-quality capture of the passenger's voice.

[1385] Voice recognition means

[1386] The server converts the collected voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API), thereby obtaining the user's speech as text data.

[1387] Natural language processing tools

[1388] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., Transformers Library). This analysis helps to understand the user's intent and the content of the question.

[1389] emotion recognition means

[1390] At the same time, the server uses an emotion recognition engine (e.g., Hugging Face emotion analysis model) to analyze the emotions from the user's speech. Emotion recognition identifies the user's state of mind based on the content of the text data.

[1391] Response generation method

[1392] The server uses a response generation AI (e.g., rinna / japanese-gpt2-medium) to generate appropriate response text based on the analysis and emotion recognition results. The generated response is tailored to the user's emotions.

[1393] Voice synthesis means

[1394] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Google Cloud Text-to-Speech API). During this conversion process, the voice tone and speed are adjusted according to the user's emotions.

[1395] Audio output means

[1396] The voice data is sent to the terminal and played back through the vehicle's speakers, allowing the user to receive a voice response.

[1397] Feedback collection methods

[1398] If a user's response is incorrect or inappropriate, they can make corrections or additions. This feedback is collected by the device's microphone and sent to the server, which receives the feedback and adds it to the system's learning database to improve the accuracy of the responses.

[1399] Specific examples

[1400] For example, if a user in a self-driving car asks "How long until the next service stop?" (in a worried voice), the system might:

[1401] 1. The device's microphone captures audio and sends the audio data to the server.

[1402] 2. The server uses a speech recognition engine to convert the voice data into text.

[1403] 3. The server analyzes the text using a natural language processing engine and understands the question.

[1404] 4. The server uses its emotion recognition engine to recognize the emotion of the utterance as "worry."

[1405] 5. The server uses a response generation AI to generate a response text that reads, "The next service area is about 30 minutes away. Please relax."

[1406] 6. When the server converts the response text into voice data using a speech synthesis engine, it responds in a gentle tone.

[1407] 7. The device plays the generated audio data from the speaker.

[1408] Prompt Sentence Examples

[1409] Input: "As a concierge AI for a self-driving vehicle, you are asked by a concerned passenger how long it will be to the next stop. Please create a response that will ease their concern."

[1410] Generate: "The next service area is about 30 minutes away. Relax."

[1411] This enables an automated conversation system that utilizes voice generation AI and an emotion recognition engine to provide flexible responses according to the user's emotional state, improving the user experience.

[1412] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1413] Step 1:

[1414] The device's microphone collects the user's speech and captures it as voice data. The input is the user's speech, and the output is voice data. This voice data is stored on the device as an electronic signal.

[1415] Step 2:

[1416] The device sends the collected voice data to the server. The input is the voice data stored on the device, and the output is the voice data transferred to the server.

[1417] Step 3:

[1418] The server uses a speech recognition engine to convert the voice data into text data. The input is voice data and the output is text data. The speech recognition engine (e.g., Google Speech-to-Text API) performs this conversion.

[1419] Step 4:

[1420] The server uses a natural language processing engine to analyze the text data and understand the user's intent. The input is text data, and the output is an analysis result that indicates the user's intent. The natural language processing engine (e.g., Transformers Library) performs this analysis.

[1421] Step 5:

[1422] The server uses an emotion recognition engine to recognize the user's emotions from the text data. The input is the text data, and the output is the emotion recognition result. The emotion recognition engine (e.g., Hugging Face emotion analysis model) performs this analysis.

[1423] Step 6:

[1424] The server generates a response text using a response generation AI based on the analysis results and emotion recognition results. The input is the analysis results and emotion recognition results, and the output is the response text. This generation is performed by the response generation AI (e.g., rinna / japanese-gpt2-medium).

[1425] Step 7:

[1426] The server uses a speech synthesis engine to convert the generated response text into audio data. The input is the response text and the output is audio data. The speech synthesis engine (e.g., Google Cloud Text-to-Speech API) performs this conversion.

[1427] Step 8:

[1428] The server sends the voice data to the terminal. The input is the voice data generated by the server, and the output is the voice data transferred to the terminal.

[1429] Step 9:

[1430] The device plays the received audio data from the speaker. The input is the audio data stored on the device, and the output is the audio played from the speaker.

[1431] Step 10:

[1432] The user hears the reply and provides feedback on the content. The input is the user's recognition and the output is the feedback speech.

[1433] Step 11:

[1434] The terminal captures the feedback voice from the user and sends it to the server. The input is the feedback voice, and the output is the feedback voice transferred to the server.

[1435] Step 12:

[1436] The server analyzes the feedback speech and uses it to improve the system. The input is the feedback speech, and the output is the system's learning and improvement results. This allows the system to generate more accurate responses.

[1437] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1438] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1439] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1440] [Fourth embodiment]

[1441] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1442] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1443] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1444] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1445] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1447] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1448] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1449] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1450] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1451] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1452] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1453] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1454] Detailed Description of the Invention

[1455] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[1456] The system mainly consists of the following components:

[1457] 1. Voice input means (terminal)

[1458] 2. Speech recognition means (server)

[1459] 3. Natural language processing means (server)

[1460] 4. Response Generation Method (Server)

[1461] 5. Speech synthesis means (server)

[1462] 6. Audio output means (terminal)

[1463] 7. Feedback Collection Method (Server)

[1464] A means of collecting user utterances

[1465] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[1466] Converting audio data to text

[1467] The server converts the collected voice data into text data using a voice recognition engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[1468] Analysis of speech content

[1469] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes the intent as "ask about the current temperature."

[1470] Generate a response

[1471] Based on the analysis results, the server uses speech generation AI to generate an appropriate response, such as "The current temperature is 25 degrees."

[1472] Response text transcription

[1473] The server converts the generated response text into voice data using a speech synthesis engine, which generates a voice that the user perceives as natural speech.

[1474] Audio output

[1475] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[1476] Gathering feedback and learning

[1477] If the user's response is inaccurate or inappropriate, they can make corrections or additions. This feedback is sent to the server and added to the system's learning database. The server uses the feedback to retrain the system to improve its performance.

[1478] Specific examples

[1479] The process will be explained as a concrete example when a user asks a robot, "What's the temperature today?"

[1480] 1. User says: "What time is it today?"

[1481] 2. The device's microphone captures the audio and sends the audio data to the server.

[1482] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[1483] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[1484] 5. The server uses a speech generation AI to generate a response text saying, "The current temperature is 25 degrees."

[1485] 6. The server uses a speech synthesis engine to convert the response text into voice data.

[1486] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[1487] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[1488] 9. The server receives the feedback and retrains the system.

[1489] This will enable natural and fluent dialogue between the robot and the user, resulting in a system that improves the user experience.

[1490] The processing flow will be explained below.

[1491] Specific processing steps of the program

[1492] Step 1:

[1493] The user speaks to the robot. At this time, the user's utterance is input as voice data, such as "What is the temperature today?"

[1494] Step 2:

[1495] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[1496] Step 3:

[1497] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[1498] Step 4:

[1499] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the user's intent: "I'm asking about the current temperature."

[1500] Step 5:

[1501] Based on the analysis results, the server calls a response generation module, which sends a request to an external API that provides temperature information. For example, the API returns the data "25 degrees."

[1502] Step 6:

[1503] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text. The generated response text is "The current temperature is 25 degrees."

[1504] Step 7:

[1505] The server inputs the generated response text into a text-to-speech engine (TTS), which converts the text data into speech data, generating speech that is easy for the user to understand.

[1506] Step 8:

[1507] The server sends the converted audio data to the device, which then plays it back through the device's speaker.

[1508] Step 9:

[1509] The user hears the robot's response. This time, the user receives the response, "The current temperature is 25 degrees."

[1510] Step 10:

[1511] If the user feels that the answer is not accurate, they can ask questions again or provide additional feedback. This feedback information is again collected by the device's microphone and sent to the server.

[1512] Step 11:

[1513] The server receives user feedback and adds it to the system's learning database, which allows the system to retrain and improve the accuracy of future responses.

[1514] This completes the process of an automated conversation system that utilizes voice generation AI.

[1515] Example 1

[1516] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1517] Conventional automated conversation systems have difficulty accurately understanding user intent and generating appropriate responses. Furthermore, they lack the ability to incorporate user feedback into learning to improve the system's accuracy, resulting in insufficient improvement in the user experience. Furthermore, the system's ability to effectively obtain external information and generate responses based on it is also limited. This leaves users unable to have satisfying conversations.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1519] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating response text using a generative AI model based on the analysis results, and means for converting the generated response text into voice data. This enables the server to generate appropriate and natural responses to user questions and improve the user experience. Furthermore, by including means for receiving feedback from the user and using it to learn and improve the system, the accuracy of the system can be improved. Furthermore, by including means for acquiring external information based on the analysis results, the server can provide more comprehensive responses.

[1520] The "means for collecting user speech as voice data" refers to a device or method for inputting the user's speech as a digital signal.

[1521] The "means for converting collected voice data into text data" refers to a technique or device that analyzes collected voice data and converts it into corresponding text data.

[1522] The "means for analyzing the converted text data and understanding the user's intent" refers to a technique or method for analyzing the text data using natural language processing technology and identifying the user's speech intent.

[1523] "Means for generating response text using a generative AI model based on analysis results" refers to techniques and methods that utilize a generative AI model to create appropriate response text based on analysis results that understand the user's intent.

[1524] The "means for converting the generated response text into voice data" refers to a technique or device that converts the generated response in text format into voice data using voice synthesis technology.

[1525] "Means for receiving user feedback and using it to learn and improve the system" refers to techniques and methods for collecting feedback information provided by users and incorporating it into the re-learning process using machine learning algorithms to improve the system and increase its accuracy.

[1526] "Means for acquiring external information based on the analysis results" refers to techniques or methods for acquiring the necessary external information from external sources such as databases or the Internet in accordance with the analyzed user's intentions.

[1527] The present invention provides an automated conversation system that utilizes speech generation AI. This system executes a series of processes to collect user utterances, analyze them, and generate responses. Specific embodiments of the system are described below.

[1528] Voice input means

[1529] When a user speaks to the robot, their voice is collected by the device's microphone. The collected voice data is sent to the server in real time. Examples of voice input methods include a smartphone or a dedicated voice input device.

[1530] Converting audio data to text

[1531] The server that receives the voice data converts the collected voice data into text data using a speech recognition engine such as the Google Speech-to-Text API. For example, a user's utterance, "What time is it today?", is generated as text data. This process includes feature extraction and phoneme recognition technology to improve the accuracy of the voice-to-text conversion.

[1532] Analysis of speech content

[1533] The converted text data is sent to the server's natural language processing (NLP) engine. NLP engines such as SpaCy and Google Cloud Natural Language API are used. This analyzes the text data and understands the user's intent. At this point, the system recognizes the intent to "ask about the current temperature."

[1534] Generate a response

[1535] Based on the analysis results, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response text. For example, a response text such as "The current temperature is 25 degrees" is generated. An example of an input prompt for the generative AI model is as follows:

[1536] "The user asked 'What's the temperature today?' Generate a response."

[1537] Response text transcription

[1538] The generated response text is converted into voice data using a speech synthesis engine such as the Google Text-to-Speech API on the server. The speech synthesis engine has phonetic conversion technology and a speech waveform generation algorithm for converting text into speech waveforms.

[1539] Audio output

[1540] The generated voice data is sent from the server to the device. It is played back as voice from the device's speaker, and the user can hear the reply, "The current temperature is 25 degrees." The voice output means can be the robot's speaker or a smartphone speaker.

[1541] Gathering feedback and learning

[1542] If a user's response is incorrect or inappropriate, they can provide corrections or additional feedback. This feedback is sent to the server and added to the system's learning database. The server then uses this feedback to retrain the system to improve its performance. Machine learning frameworks such as TensorFlow are used for retraining.

[1543] Specific example explanation

[1544] As a concrete example, if a user asks a robot "What time is it today?", the process is as follows:

[1545] 1. User says: "What time is it today?"

[1546] 2. The device's microphone captures the audio and sends the audio data to the server.

[1547] 3. The server uses the Google Speech-to-Text API to convert the voice data into text: "What time is it today?"

[1548] 4. The server uses an NLP engine to parse the text and understand the intent: asking about the current temperature.

[1549] 5. The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[1550] 6. The server uses the Google Text-to-Speech API to convert the response text into audio data.

[1551] 7. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[1552] 8. The user listens to the response and possibly asks further questions or provides additional feedback.

[1553] 9. The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[1554] This series of processes enables the user to have natural and fluent conversations with the robot, resulting in a system that improves the user experience.

[1555] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1556] Step 1:

[1557] User says: "What time is it today?"

[1558] Input: User's spoken utterance

[1559] Output: Digital audio data captured by the device's microphone

[1560] How it works: The user speaks to the robot, and their voice is captured through the device's microphone. At this stage, the voice is temporarily stored as digital data in a buffer.

[1561] Step 2:

[1562] The device's microphone captures the audio and sends the audio data to the server.

[1563] Input: Digital audio data

[1564] Output: Digital audio data sent to the server

[1565] Operation: The terminal transmits the captured audio data to the server in real time. The audio data is transmitted over the network using the TCP / IP protocol.

[1566] Step 3:

[1567] The server converts the audio data into text using the Google Speech-to-Text API.

[1568] Input: Digital audio data sent to the server

[1569] Output: Text data "What's the temperature today?"

[1570] How it works: The server passes the received audio data to the Google Speech-to-Text API, which converts the audio data into phonemes and generates the corresponding text data based on them.

[1571] Step 4:

[1572] The server uses an NLP engine to parse the text and understand the intent of asking about the current temperature.

[1573] Input: Text data "What's the temperature today?"

[1574] Output: Parsing the intent "Ask about the current temperature"

[1575] How it works: The text data is sent to the server's NLP engine (e.g., SpaCy or Google Cloud Natural Language API), which uses intent recognition algorithms to analyze the user's question intent.

[1576] Step 5:

[1577] The server uses a generative AI model (e.g., GPT-4) to generate a response text such as "The current temperature is 25 degrees."

[1578] Input: Parsed intent "Ask about the current temperature", prompt: "The user asked 'What's the temperature today?' Generate a response."

[1579] Output: Response text "The current temperature is 25 degrees."

[1580] How it works: The server provides the generative AI model with an input prompt and generates an appropriate response text based on the intent. The generative AI model creates this response based on a pre-trained dataset.

[1581] Step 6:

[1582] The server converts the response text into audio data using the Google Text-to-Speech API.

[1583] Input: Response text "The current temperature is 25 degrees"

[1584] Output: Audio data saying "The current temperature is 25 degrees"

[1585] How it works: The generated response text is sent to the server's Google Text-to-Speech API, where phonetic conversion from text to speech and generation of speech waveforms are performed, resulting in audio data.

[1586] Step 7:

[1587] The device plays the generated audio data over the speaker: "The current temperature is 25 degrees."

[1588] Input: Speech data saying "The current temperature is 25 degrees"

[1589] Output: Voice played through speaker: "The current temperature is 25 degrees."

[1590] Operation: Audio data is sent from the server to the device and output to the user as audio through the device's speaker.

[1591] Step 8:

[1592] The user listens to the response and may ask further questions or provide additional feedback.

[1593] Input: "The current temperature is 25 degrees"

[1594] Output: User feedback

[1595] How it works: The user listens to the reply and, if necessary, enters corrections or additional feedback into the terminal, which is then sent to the server.

[1596] Step 9:

[1597] The server receives the feedback, stores it in a database, and uses TensorFlow for retraining.

[1598] Input: Feedback data received from users

[1599] Output: Improved model

[1600] How it works: Feedback data is stored on a server and the model is retrained using a machine learning framework such as TensorFlow, improving the accuracy of the system and the user experience.

[1601] (Application example 1)

[1602] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1603] Conventional autonomous vehicles have difficulty in naturally interacting with users, requiring drivers and passengers to perform complex operations to set destinations, obtain current location information, and receive advice on safe driving. Furthermore, there has been a lack of easy ways to obtain external information in real time, such as weather information notifications, which has hindered the user experience of autonomous driving.

[1604] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1605] In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intent, means for generating a response text based on the analysis result, means for converting the generated response text into voice data, means for outputting the converted voice data to the user, and means for functioning as a voice assistant in an autonomous vehicle and interactively setting a destination, providing current location information, notifying weather information, and providing advice on safe driving. This enables multifunctional interaction through natural conversation with the user in an autonomous vehicle.

[1606] A "means for collecting user speech as audio data" is a device or method that captures the user's speech and stores it as digital audio data.

[1607] The "means for converting collected voice data into text data" refers to a process of converting voice data into corresponding character strings using voice recognition technology.

[1608] "Means for analyzing the converted text data and understanding the user's intent" refers to a function that uses natural language processing technology to analyze the text data and understand the user's intent and questions from its content.

[1609] The "means for generating a response text based on the analysis results" is a process for generating an appropriate response content in response to the analyzed user intent.

[1610] The "means for converting the generated response text into voice data" is a voice synthesis technology for generating natural and fluent voice from text.

[1611] The "means for outputting the converted voice data to the user" refers to a device or method for making the generated voice data audible to the user through a speaker or the like.

[1612] "A means that functions as a voice assistant within an autonomous vehicle and interactively provides destination setting, current location information, weather information, and advice on safe driving" refers to a system or software that can interact with the user via voice within the environment of the autonomous vehicle and respond to various information and instructions.

[1613] System Program Overview

[1614] A system for implementing this invention includes the following major components:

[1615] 1. Voice input means (terminal)

[1616] 2. Speech recognition means (server)

[1617] 3. Natural language processing means (server)

[1618] 4. Response Generation Method (Server)

[1619] 5. Speech synthesis means (server)

[1620] 6. Audio output means (terminal)

[1621] 7. Feedback Collection Method (Server)

[1622] Hardware and Software

[1623] The system uses a terminal equipped with a microphone and speaker to collect the user's speech in real time. The server uses the following software to process and calculate the data:

[1624] Speech recognition: Use the speech_recognition library to convert voice data into text data.

[1625] Natural language processing and response generation: Uses OpenAI's API to analyze user intent and generate appropriate response text.

[1626] Speech synthesis: Use the gtts library to convert response text into voice data.

[1627] Audio output: The generated audio data is played through the speaker using the playsound library.

[1628] Feedback collection: Receive feedback from users and use it to train the system.

[1629] Specific examples

[1630] User questions and how to handle them

[1631] For example, consider the case where a user says, "How much battery power is currently remaining in my vehicle?" The device's microphone captures the voice and sends the voice data to the server. The server then uses a speech recognition engine to convert this voice data into text. The server then analyzes this text with a natural language processing engine to understand the intent of the user's question. Based on the analysis results, a response generation engine generates a response text such as "The current battery power remaining is 80%." The generated response text is then converted into voice data using a speech synthesis engine and sent to the device. The device's speaker plays this voice data, conveying the response to the user.

[1632] Prompt Sentence Examples

[1633] Specific prompts are used to guide the generative AI model to generate appropriate responses. For example, here are some examples of prompts that can be used when a user asks:

[1634] Answer the following question: "How much battery power is left in my current vehicle?"

[1635] Based on this prompt, the system generates an appropriate response and outputs it as voice data.

[1636] Feedback collection and learning

[1637] If a user provides feedback on a response, that feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve the system's performance. For example, if a user provides feedback indicating dissatisfaction with a response, the system will adjust to provide better responses to similar questions in the future.

[1638] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1639] Step 1:

[1640] A means of collecting user utterances

[1641] When a user speaks in an autonomous vehicle, the device's microphone captures the voice. This voice data is collected in digital form and sent to a server. The input is the user's voice, and the output is digital voice data.

[1642] Step 2:

[1643] A means of converting audio data into text data

[1644] The server converts the received voice data into text data using a speech recognition engine (speech_recognition library). In this process, the voice waveform data is analyzed and the corresponding text is generated. The input is digital voice data, and the output is text data.

[1645] Step 3:

[1646] A means of analyzing text data and understanding user intent

[1647] The server analyzes the converted text data using a natural language processing engine (OpenAI API) to understand the user's intent. This analysis identifies what the user is looking for based on the content of the input text. The input is text data, and the output is the user's intent (for example, checking the remaining battery level).

[1648] Step 4:

[1649] A way to generate response text

[1650] Based on the analysis results, the server uses a response generation engine (OpenAI API) to generate an appropriate response text. The generative AI model creates a response text based on the prompt text. The input is the user's intention, and the output is the response text. An example of a specific prompt text is "Please answer the following question: 'How much battery is left in your current vehicle?'"

[1651] Step 5:

[1652] A means of converting response text into audio data

[1653] The server converts the generated response text into voice data using a speech synthesis engine (gtts library). Natural and fluent speech is generated from the text. The input is the response text, and the output is voice data.

[1654] Step 6:

[1655] A means for outputting audio data to the user

[1656] The server sends the generated voice data to the device and plays it from the device's speaker, allowing the user to hear the reply. The input is voice data, and the output is voice played from the speaker.

[1657] Step 7:

[1658] Gathering feedback and learning

[1659] Users can provide feedback on responses. For example, if a response is inappropriate, the user can offer corrections or suggestions for clarification. This feedback is sent to the server and added to the system's learning database. The server uses this feedback to retrain and improve system performance. The input is the user's feedback, and the output is an updated learning database.

[1660] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1661] Detailed Description of the Invention

[1662] The present invention provides an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. This system not only collects and analyzes user utterances to generate responses, but also recognizes the user's emotions and responds based on those emotions. Specific embodiments of the system are described below.

[1663] The system mainly consists of the following components:

[1664] 1. Voice input means (terminal)

[1665] 2. Speech recognition means (server)

[1666] 3. Natural language processing means (server)

[1667] 4. Emotion Recognition Method (Server)

[1668] 5. Response Generation Method (Server)

[1669] 6. Speech synthesis means (server)

[1670] 7. Audio output means (terminal)

[1671] 8. Feedback Collection Method (Server)

[1672] A means of collecting user utterances

[1673] When a user speaks to the robot, their voice is collected by the device's microphone and sent to a server in real time.

[1674] Converting audio data to text

[1675] The server converts the collected voice data into text data using an automatic speech recognition (ASR) engine. For example, a user's utterance, "What time is it today?", is generated as text data.

[1676] Analysis of speech content

[1677] The server analyzes the text data using a natural language processing (NLP) engine to understand the user's intent. Here, the system recognizes that the user is asking about the current temperature.

[1678] Emotion recognition

[1679] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from their speech. For example, the utterance "What's the temperature today?" is recognized as "excited" or "calm."

[1680] Generate a response

[1681] Based on the analysis results and emotion recognition results, the server uses speech generation AI to generate an appropriate response text. For example, it might generate a response such as "The current temperature is 25 degrees." The response text is adjusted according to the user's emotions. If the user is excited, the response will be adjusted to sound calm.

[1682] Response text transcription

[1683] The server converts the generated response text into voice data using a text-to-speech engine (TTS). At this time, the tone and speed of the voice are adjusted based on the user's emotions. For example, if the user is calm, the response will be in a calm tone.

[1684] Audio output

[1685] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the response adjusted with the appropriate tone and speed.

[1686] Gathering feedback and learning

[1687] If the user's response is incorrect or inappropriate, they can make corrections or clarifications. This feedback is collected by the device's microphone and sent back to the server.

[1688] The server receives the feedback and adds it to the system's learning database, which allows the system to retrain itself and improve the accuracy of future responses.

[1689] Specific examples

[1690] Let us take a concrete example of the process when a user asks the robot in an excited voice, "What's the temperature today?"

[1691] 1. User says: "What's the temperature today?" (in an excited voice)

[1692] 2. The device's microphone captures the audio and sends the audio data to the server.

[1693] 3. The server uses a speech recognition engine to convert the voice data into text: "What time is it today?"

[1694] 4. The server uses an NLP engine to parse the text and understand the intent: to ask about the current temperature.

[1695] 5. The server uses an emotion recognition engine to recognize the emotion of the utterance as "excited."

[1696] 6. The server calls the response generation module based on the analysis results and emotion recognition results, and the speech generation AI generates the response text, "The current temperature is 25 degrees."

[1697] 7. The server converts the response text into voice data using a speech synthesis engine, responding in a calm tone to an excited user.

[1698] 8. The device plays the generated audio data through the speaker: "The current temperature is 25 degrees."

[1699] 9. The user listens to the response and possibly asks further questions or provides additional feedback.

[1700] 10. The server receives user feedback and retrains the system.

[1701] This will enable automated conversation systems that utilize voice generation AI and emotion recognition engines to achieve more natural and flexible dialogue, improving the user experience.

[1702] The processing flow will be explained below.

[1703] Detailed Description of the Invention

[1704] The specific processing steps of a conversation system incorporating an emotion recognition engine will be described below.

[1705] Step 1:

[1706] The user speaks to the robot, for example, saying, "What's the temperature today?"

[1707] Step 2:

[1708] The device's microphone captures the user's speech and collects it as voice data, which is then sent to a server in real time.

[1709] Step 3:

[1710] The voice data received by the server is input into a speech recognition engine (ASR) and converted into text data. Specifically, the text data generated is "What time is it today?"

[1711] Step 4:

[1712] The server passes the text data to a natural language processing (NLP) engine, which performs a semantic analysis of the text, recognizing the intent that the user is asking about the current temperature.

[1713] Step 5:

[1714] At the same time, the server passes the voice data to an emotion recognition engine to analyze the user's emotions, which then recognizes the user's speech as "excited" or "calm."

[1715] Step 6:

[1716] Based on the analysis results (intent and emotion), the server calls the response generation module. This module sends a request to an external API to obtain temperature information. For example, the API returns the data "25 degrees."

[1717] Step 7:

[1718] Based on the external information acquired by the server, the speech generation AI generates an appropriate response text, which is "The current temperature is 25 degrees."

[1719] Step 8:

[1720] The server inputs the response text into a text-to-speech engine (TTS) and converts the text data into voice data. During voice synthesis, the tone and speed of the voice are adjusted according to the user's emotions. For example, if the user is excited, the server will respond in a calm tone.

[1721] Step 9:

[1722] The generated voice data is sent to the terminal and played back through the robot's speaker, allowing the user to hear the reply, "The current temperature is 25 degrees."

[1723] Step 10:

[1724] If the user feels that the answer is not accurate, they can ask the question again or provide additional feedback, which is again collected by the device's microphone and sent to the server.

[1725] Step 11:

[1726] The server receives user feedback and adds it to the system's learning database, allowing the system to retrain based on the feedback and improve the accuracy of future responses.

[1727] This enables speech generation AI systems incorporating emotion recognition engines to enable more natural and emotionally appropriate dialogue, improving the user experience.

[1728] Example 2

[1729] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] Conventional automated conversation systems can collect user utterances, convert them into text data, and generate responses. However, they are unable to interpret the user's emotions and respond appropriately, resulting in a lack of naturalness in the dialogue and a poor user experience. Furthermore, it is difficult to utilize feedback to continuously improve the system and respond to diverse user needs.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1732] In this invention, the server includes means for recognizing emotions from user utterances, means for generating response text based on the analysis results and emotion recognition results, and means for receiving feedback from the user and using it to learn and improve the system, thereby enabling more natural and effective dialogue that takes user emotions into consideration and allowing the system to learn and improve accuracy.

[1733] "User utterance" refers to a linguistic expression that a user verbally inputs to the system.

[1734] "Voice data" refers to the user's speech converted into a digital signal.

[1735] A "collection means" is a device or program that captures a user's voice data and stores it for processing.

[1736] "Text data" refers to character string information obtained by analyzing voice data.

[1737] "Means for converting" refers to devices or software for converting voice data into text data.

[1738] "Means of analysis" refers to machine learning models and programs that break down and analyze text data to understand its content and intent.

[1739] "User intent" refers to the information or request that a user wants to convey through speech.

[1740] "Means for recognizing emotions" refers to algorithms or programs that determine the emotions expressed by the user at that time from their speech.

[1741] "Analysis results" refers to the information obtained after interpreting the content and intent of text data.

[1742] The "emotion recognition result" is the classification result of emotions analyzed from the user's speech.

[1743] "Response text" is a character string representing the response content corresponding to the user's utterance.

[1744] "Generating means" refers to a device or program that creates a response text based on the analysis results and emotion recognition results.

[1745] "Means for converting into voice data" refers to the device or software that converts the response text back into voice form.

[1746] "Output means" refers to a device or program that plays the converted audio data on a speaker or the like so that the user can hear it.

[1747] "Feedback" refers to the evaluation or opinion that a user gives to the system's response.

[1748] "Means used to learn and improve the system" refers to machine learning algorithms and programs that incorporate user feedback to improve the system's performance and response accuracy.

[1749] This invention provides an automatic conversation system that utilizes a voice generation AI and an emotion recognition engine. Specifically, it is designed not only to collect user utterances, analyze them, and generate responses, but also to recognize the user's emotions and respond based on those emotions.

[1750] The system includes the following components:

[1751] 1. Voice input means (terminal)

[1752] 2. Speech recognition means (server)

[1753] 3. Natural language processing means (server)

[1754] 4. Emotion Recognition Method (Server)

[1755] 5. Response Generation Method (Server)

[1756] 6. Speech synthesis means (server)

[1757] 7. Audio output means (terminal)

[1758] 8. Feedback Collection Method (Server)

[1759] The device is equipped with a microphone that collects the user's voice. This microphone captures the voice spoken by the user and converts it into voice data. For example, if a user speaks to the robot, "What time is it today?", the voice is collected by the microphone.

[1760] The device sends the collected voice data to the server in real time. A secure protocol (e.g., HTTPS) is used for this transmission. The server then converts the received voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text). For example, the voice "What's the temperature today?" is output as the text "What's the temperature today?"

[1761] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., GPT-3). During this process, it understands that the intent of the text is "asking about the current temperature." At the same time, the server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze the emotion from the user's speech. For example, the utterance "What's the temperature today?" is recognized as "excited."

[1762] Based on the analysis results and emotion recognition results, the server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text. For example, the response text generated might be, "The current temperature is 25 degrees." The tone of the response is adjusted according to the user's emotion. If the user is excited, the response will be set to a calm tone.

[1763] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Amazon Polly). The tone and speed of the voice are also adjusted based on the user's emotions. For example, if the response is made in a calm tone, the server will respond in a calm tone, saying, "The current temperature is 25 degrees."

[1764] The terminal receives the generated voice data and plays it back through the speaker. The user can hear the response, adjusted with the appropriate tone and speed. An example of a specific prompt sentence is the following dialogue when the user asks, "What time is it today?"

[1765] If the user adds or corrects a reply, that feedback is collected by the device's microphone and sent back to the server. The server receives the feedback, adds it to the system's learning database, and retrains it to improve the accuracy of future replies. This allows the system to continuously improve and provide more natural and accurate dialogue.

[1766] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1767] Step 1:

[1768] The user says, "What's the temperature today?"

[1769] Input: User's speech (voice data)

[1770] Output: Audio data captured by the device's microphone

[1771] Specific operation: The device's microphone captures the user's voice and obtains the voice data.

[1772] Step 2:

[1773] The device transmits the audio data to the server in real time.

[1774] Input: Audio data captured by the device's microphone

[1775] Output: Audio data sent to the server

[1776] What happens: The device sends the captured audio data to the server via a secure protocol (e.g., HTTPS).

[1777] Step 3:

[1778] The server converts the voice data into text data using a voice recognition engine.

[1779] Input: Transmitted audio data

[1780] Output: The converted text data (e.g. "What time is it today?")

[1781] Specific operation: The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data.

[1782] Step 4:

[1783] The server uses a natural language processing engine to analyze the text data and understand the user's intent.

[1784] Input: Converted text data

[1785] Output: Analysis results indicating the user's intent (e.g., "I'm asking for the current temperature")

[1786] Specific operation: The server uses a natural language processing (NLP) engine (e.g., GPT-3) to analyze text data and understand the user's intent.

[1787] Step 5:

[1788] The server uses an emotion recognition engine to recognize emotions from the user's speech.

[1789] Input: Converted text data

[1790] Output: Recognition result indicating the user's emotion (e.g. excited)

[1791] Specific operation: The server uses an emotion recognition engine (e.g., IBM Watson Tone Analyzer) to analyze emotions from the user's speech.

[1792] Step 6:

[1793] The server generates a response text based on the analysis results and emotion recognition results.

[1794] Input: Analysis results and emotion recognition results that indicate the user's intentions

[1795] Output: Generated response text (e.g. "The current temperature is 25 degrees")

[1796] Specific operation: The server uses a response generation AI (e.g., GPT-3) to generate an appropriate response text.

[1797] Step 7:

[1798] The server converts the generated response text into audio data.

[1799] Input: Generated response text

[1800] Output: Converted audio data

[1801] What happens: The server uses a speech synthesis engine (e.g., Amazon Polly) to convert the response text into speech data, adjusting the tone and speed of the speech.

[1802] Step 8:

[1803] The server sends the audio data to the terminal.

[1804] Input: Converted audio data

[1805] Output: Audio data sent to the device

[1806] Specific operation: The server sends the audio data to the device using a secure protocol (e.g. HTTPS).

[1807] Step 9:

[1808] The audio data received by the device is output from the speaker.

[1809] Input: Transmitted audio data

[1810] Output: What the user hears (e.g., "The current temperature is 25 degrees")

[1811] What happens: The response is played through the device speaker with a controlled voice tone and speed.

[1812] Step 10:

[1813] The user provides feedback on the response.

[1814] Input: User feedback (voice data)

[1815] Output: Feedback audio data captured by the device's microphone

[1816] Specific operation: The device's microphone captures the user's feedback and obtains the audio data.

[1817] Step 11:

[1818] The terminal transmits the feedback voice data to the server.

[1819] Input: Captured feedback audio data

[1820] Output: Feedback audio data sent to the server

[1821] Specific operation: The device sends feedback audio data to the server via a secure protocol (e.g., HTTPS).

[1822] Step 12:

[1823] The server receives the feedback, adds it to the system's learning database, and retrains it.

[1824] Input: Submitted feedback audio data

[1825] Output: Updated training database and improved system

[1826] Specific operation: The server analyzes the feedback, adds it to the system's learning database, and performs re-training to improve response accuracy.

[1827] (Application example 2)

[1828] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1829] Conventional automated conversation systems can analyze user utterances and generate responses, but they face the challenge of generating appropriate responses that take the user's emotions into account. Furthermore, for users to safely and naturally converse in an autonomous vehicle while driving or traveling, responses that reflect the user's emotional state are required. This calls for a more advanced dialogue system.

[1830] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterances as voice data, means for converting the collected voice data into text data, means for analyzing the converted text data and understanding the user's intention, means for recognizing emotions from the user's utterance, means for generating a response text based on the analysis result and the emotion recognition result, means for converting the generated response text into voice data, and means for outputting the converted voice data to the user. This enables an appropriate response according to the user's emotions.

[1831] "User" refers to a person who uses the system.

[1832] "Utterance" refers to words spoken by a user.

[1833] "Voice data" refers to data that is an electronic recording of a user's speech.

[1834] "Collection means" refers to a device or method for acquiring audio data.

[1835] "Text data" refers to data that has been converted from voice data into text information.

[1836] "Conversion means" refers to a device or method for converting audio data into text data.

[1837] "Analysis means" refers to a device or method that analyzes text data and understands the user's intent and content.

[1838] "Emotion recognition means" refers to a device or method for identifying emotions from a user's speech.

[1839] "Response generation means" refers to a device or method that generates an appropriate response based on the analysis results and emotion recognition results.

[1840] "Speech synthesis means" refers to a device or method that converts generated text responses into speech.

[1841] "Audio output means" refers to a device or method for reproducing audio data and allowing the user to listen to it.

[1842] "Feedback collection means" refers to a device or method for collecting corrections or supplemental information from users.

[1843] An "autonomous vehicle" is a vehicle that operates automatically without a human driver.

[1844] This invention is an automated conversation system that utilizes a speech generation AI and an emotion recognition engine. A specific embodiment for collecting user utterances in an autonomous vehicle and generating appropriate responses is described below.

[1845] System Overview

[1846] The system mainly consists of the following components:

[1847] 1. Voice input means (terminal)

[1848] 2. Speech recognition means (server)

[1849] 3. Natural language processing means (server)

[1850] 4. Emotion Recognition Method (Server)

[1851] 5. Response Generation Method (Server)

[1852] 6. Speech synthesis means (server)

[1853] 7. Audio output means (terminal)

[1854] 8. Feedback Collection Method (Server)

[1855] Voice input means

[1856] The device includes a microphone placed inside the autonomous vehicle to collect the user's voice, and the microphone is equipped with noise cancellation to ensure high-quality capture of the passenger's voice.

[1857] Voice recognition means

[1858] The server converts the collected voice data into text data using a voice recognition engine (e.g., Google Speech-to-Text API), thereby obtaining the user's speech as text data.

[1859] Natural language processing tools

[1860] The server analyzes the converted text data using a natural language processing (NLP) engine (e.g., Transformers Library). This analysis helps to understand the user's intent and the content of the question.

[1861] emotion recognition means

[1862] At the same time, the server uses an emotion recognition engine (e.g., Hugging Face emotion analysis model) to analyze the emotions from the user's speech. Emotion recognition identifies the user's state of mind based on the content of the text data.

[1863] Response generation method

[1864] The server uses a response generation AI (e.g., rinna / japanese-gpt2-medium) to generate appropriate response text based on the analysis and emotion recognition results. The generated response is tailored to the user's emotions.

[1865] Voice synthesis means

[1866] The server converts the generated response text into voice data using a speech synthesis engine (e.g., Google Cloud Text-to-Speech API). During this conversion process, the voice tone and speed are adjusted according to the user's emotions.

[1867] Audio output means

[1868] The voice data is sent to the terminal and played back through the vehicle's speakers, allowing the user to receive a voice response.

[1869] Feedback collection methods

[1870] If a user's response is incorrect or inappropriate, they can make corrections or additions. This feedback is collected by the device's microphone and sent to the server, which receives the feedback and adds it to the system's learning database to improve the accuracy of the responses.

[1871] Specific examples

[1872] For example, if a user in a self-driving car asks "How long until the next service stop?" (in a worried voice), the system might:

[1873] 1. The device's microphone captures audio and sends the audio data to the server.

[1874] 2. The server uses a speech recognition engine to convert the voice data into text.

[1875] 3. The server analyzes the text using a natural language processing engine and understands the question.

[1876] 4. The server uses its emotion recognition engine to recognize the emotion of the utterance as "worry."

[1877] 5. The server uses a response generation AI to generate a response text that reads, "The next service area is about 30 minutes away. Please relax."

[1878] 6. When the server converts the response text into voice data using a speech synthesis engine, it responds in a gentle tone.

[1879] 7. The device plays the generated audio data from the speaker.

[1880] Prompt Sentence Examples

[1881] Input: "As a concierge AI for a self-driving vehicle, you are asked by a concerned passenger how long it will be to the next stop. Please create a response that will ease their concern."

[1882] Generate: "The next service area is about 30 minutes away. Relax."

[1883] This enables an automated conversation system that utilizes voice generation AI and an emotion recognition engine to provide flexible responses according to the user's emotional state, improving the user experience.

[1884] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1885] Step 1:

[1886] The device's microphone collects the user's speech and captures it as voice data. The input is the user's speech, and the output is voice data. This voice data is stored on the device as an electronic signal.

[1887] Step 2:

[1888] The device sends the collected voice data to the server. The input is the voice data stored on the device, and the output is the voice data transferred to the server.

[1889] Step 3:

[1890] The server uses a speech recognition engine to convert the voice data into text data. The input is voice data and the output is text data. The speech recognition engine (e.g., Google Speech-to-Text API) performs this conversion.

[1891] Step 4:

[1892] The server uses a natural language processing engine to analyze the text data and understand the user's intent. The input is text data, and the output is an analysis result that indicates the user's intent. The natural language processing engine (e.g., Transformers Library) performs this analysis.

[1893] Step 5:

[1894] The server uses an emotion recognition engine to recognize the user's emotions from the text data. The input is the text data, and the output is the emotion recognition result. The emotion recognition engine (e.g., Hugging Face emotion analysis model) performs this analysis.

[1895] Step 6:

[1896] The server generates a response text using a response generation AI based on the analysis results and emotion recognition results. The input is the analysis results and emotion recognition results, and the output is the response text. This generation is performed by the response generation AI (e.g., rinna / japanese-gpt2-medium).

[1897] Step 7:

[1898] The server uses a speech synthesis engine to convert the generated response text into audio data. The input is the response text and the output is audio data. The speech synthesis engine (e.g., Google Cloud Text-to-Speech API) performs this conversion.

[1899] Step 8:

[1900] The server sends the voice data to the terminal. The input is the voice data generated by the server, and the output is the voice data transferred to the terminal.

[1901] Step 9:

[1902] The device plays the received audio data from the speaker. The input is the audio data stored on the device, and the output is the audio played from the speaker.

[1903] Step 10:

[1904] The user hears the reply and provides feedback on the content. The input is the user's recognition and the output is the feedback speech.

[1905] Step 11:

[1906] The terminal captures the feedback voice from the user and sends it to the server. The input is the feedback voice, and the output is the feedback voice transferred to the server.

[1907] Step 12:

[1908] The server analyzes the feedback speech and uses it to improve the system. The input is the feedback speech, and the output is the system's learning and improvement results. This allows the system to generate more accurate responses.

[1909] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1910] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1911] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1912] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1913] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1914] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1915] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1916] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1917] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1918] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1919] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1920] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1921] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1922] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1923] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1924] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1925] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1926] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1927] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1928] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1929] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1930] The following is further disclosed regarding the above embodiment.

[1931] (Claim 1)

[1932] A means for collecting user utterances as voice data;

[1933] A means for converting the collected voice data into text data;

[1934] A means for analyzing the converted text data and understanding the user's intent;

[1935] means for generating a response text based on the analysis result;

[1936] means for converting the generated response text into voice data;

[1937] The system includes means for outputting the converted audio data to a user.

[1938] (Claim 2)

[1939] 10. The system of claim 1, including means for receiving feedback from users and using it to learn and improve the system.

[1940] (Claim 3)

[1941] 10. The system according to claim 1, further comprising means for acquiring external information based on the analysis result.

[1942] (Claim 4)

[1943] 10. The system of claim 1, further comprising: means for generating a response text based on the obtained external information.

[1944] "Example 1"

[1945] (Claim 1)

[1946] A means for collecting user utterances as voice data;

[1947] A means for converting the collected voice data into text data;

[1948] A means for analyzing the converted text data and understanding the user's intent;

[1949] A means for generating a response text using a generative AI model based on the analysis result;

[1950] means for converting the generated response text into voice data;

[1951] The system includes means for outputting the converted audio data to a user.

[1952] (Claim 2)

[1953] 10. The system of claim 1, including means for receiving feedback from users and using it to learn and improve the system.

[1954] (Claim 3)

[1955] 10. The system according to claim 1, further comprising means for acquiring external information based on the analysis result.

[1956] "Application Example 1"

[1957] (Claim 1)

[1958] A means for collecting user utterances as voice data;

[1959] A means for converting the collected voice data into text data;

[1960] A means for analyzing the converted text data and understanding the user's intent;

[1961] means for generating a response text based on the analysis result;

[1962] means for converting the generated response text into voice data;

[1963] means for outputting the converted audio data to a user;

[1964] A system that functions as a voice assistant within an autonomous vehicle and includes means to interactively set destinations, provide current location information, notify weather information, and provide advice on safe driving.

[1965] (Claim 2)

[1966] 10. The system of claim 1, including means for receiving feedback from users and using it to learn and improve the system.

[1967] (Claim 3)

[1968] 10. The system according to claim 1, further comprising means for acquiring external information based on the analysis result.

[1969] "Example 2: Combining Emotion Engines"

[1970] (Claim 1)

[1971] A means for collecting user utterances as voice data;

[1972] A means for converting the collected voice data into text data;

[1973] A means for analyzing the converted text data and understanding the user's intent;

[1974] means for recognizing emotions from a user's speech;

[1975] means for generating a response text based on the analysis result and the emotion recognition result;

[1976] means for converting the generated response text into voice data;

[1977] The system includes means for outputting the converted audio data to a user.

[1978] (Claim 2)

[1979] 10. The system of claim 1, including means for receiving feedback from users and using it to learn and improve the system.

[1980] (Claim 3)

[1981] 10. The system according to claim 1, further comprising means for acquiring external information based on the analysis result.

[1982] "Application example 2 when combining emotion engines"

[1983] (Claim 1)

[1984] A means for collecting user utterances as voice data;

[1985] A means for converting the collected voice data into text data;

[1986] A means for analyzing the converted text data and understanding the user's intent;

[1987] means for recognizing emotions from a user's speech;

[1988] means for generating a response text based on the analysis result and the emotion recognition result;

[1989] means for converting the generated response text into voice data;

[1990] The system includes means for outputting the converted audio data to a user.

[1991] (Claim 2)

[1992] 10. The system of claim 1, including means for receiving feedback from users and using it to learn and improve the system.

[1993] (Claim 3)

[1994] 10. The system according to claim 1, further comprising means for acquiring external information based on the analysis result. [Explanation of symbols]

[1995] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting user utterances as voice data; A means for converting the collected voice data into text data; A means for analyzing the converted text data and understanding the user's intent; means for generating a response text based on the analysis result; means for converting the generated response text into voice data; The system includes means for outputting the converted audio data to a user.

2. 10. The system of claim 1, further comprising means for receiving feedback from users and using it to learn and improve the system.

3. The system according to claim 1 , further comprising means for acquiring external information based on the analysis result.

4. 10. The system of claim 1, further comprising: means for generating a response text based on the obtained external information.

Images