System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a voice input of a user; means for converting the voice input into digital data; means for transmitting the digital data to a server; means including a speech recognition engine for converting the digital data into text data; means including a text-to-speech engine for analyzing the text data, understanding a user's intention, and acquiring information from an external service based on the user's intention; means for generating a response message in a natural language based on the acquired information; means for converting the response message into voice data; and means for transmitting the voice data to a terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional AI systems have low speech recognition accuracy and low natural language processing accuracy when using voice commands, making it difficult to accurately understand and execute user intent. Additionally, multiple processing steps (voice input, speech recognition, natural language processing, information acquisition, response generation, and voice output) increase processing time and slow response speeds. Furthermore, improper compression, encoding, decoding, and decompression of voice data can degrade data quality, potentially adversely affecting overall system performance. [Means for solving the problem]

[0005] The present invention provides a system including: means for receiving a user's voice input and converting the voice input into digital data; means for transmitting the digital data to a server; means including a speech recognition engine for converting the digital data into text data; natural language processing means for analyzing the text data and understanding the user's intent; means for acquiring information from an external service based on the user's intent; means for generating a natural language response message from the acquired information; means including a text-to-speech engine for converting the response message into voice data; means for transmitting the voice data to a terminal; and means for playing the voice data. The system further includes means for starting reception of the voice input in response to a trigger from the user's voice command, and means for compressing, encoding, decoding, and decompressing the voice data, thereby improving the efficiency and accuracy of processing. This makes it possible to provide a fast and accurate response to the user's voice command and optimize overall system performance.

[0006] "Voice input" refers to the voice signals spoken by the user, and is the data that the system receives and processes.

[0007] "Digital data" refers to data obtained by converting analog audio signals into digital format and transmitted to the server.

[0008] "Server" means a centralized computing resource that receives, analyzes, and processes digital data sent by users.

[0009] A "voice recognition engine" is a technology that converts digital data into text data and is used to analyze the content of a user's speech.

[0010] "Text data" refers to text information generated from voice input by a voice recognition engine, and is the subject of natural language processing.

[0011] "Natural language processing" is a technology that analyzes text data to understand the user's intent and is used to determine the appropriate task.

[0012] An "external service" is an external information providing system that the server uses to obtain information based on the user's intentions.

[0013] A "response message" is a message in natural language format that is generated based on the acquired information and that contains the content to be conveyed to the user.

[0014] A "text-to-speech engine" is a technology that converts text data into voice data, and is used to convey a response message to the user as voice.

[0015] "Audio data" means data in audio format generated by a text-to-speech engine and transmitted to a terminal for playback.

[0016] A "terminal" is a device that provides an interface between a user and a system and is used to receive voice input and play voice data.

[0017] "Compression" refers to the process of reducing the size of audio data, and is performed to improve communication efficiency.

[0018] "Encoding" is the process of converting data into a different format for more efficient communication or storage.

[0019] "Decoding" is the process of returning encoded data to its original form so that it can be properly handled as digital data.

[0020] "Decompression" is the process of returning compressed data to its original size so that it can be played back correctly as audio data.

[0021] A "trigger" is an utterance by the user that serves as a trigger for starting a specific action and indicates the start of receiving voice input. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] System Overview

[0044] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[0045] System configuration

[0046] This system mainly consists of a terminal and a server.

[0047] Terminal

[0048] The terminal receives voice input from the user and records, converts, compresses, and encodes the voice. It includes a microphone, a processor, and network communication functions. The terminal transmits the voice data to the server and receives and plays back response voice data from the server.

[0049] server

[0050] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from an external service and generates a response message in natural language. The response message is converted into voice data using a text-to-speech engine and sent to the terminal.

[0051] System Operation

[0052] The operation of the system of the present invention will now be described in detail.

[0053] Brief Description

[0054] 1. Receiving voice input (terminal)

[0055] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[0056] 2. Sending audio data (terminal)

[0057] The digital data is transmitted over a network to a server.

[0058] 3. Speech recognition (server)

[0059] The server decodes and decompresses the received voice data and converts it into text data using a voice recognition engine.

[0060] 4. Natural Language Processing (Server)

[0061] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, obtaining weather information).

[0062] 5. Information Acquisition (Server)

[0063] The server calls external services such as weather forecast APIs to obtain the latest weather information.

[0064] 6. Response Generation (Server)

[0065] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated.

[0066] 7. Voice data generation (server)

[0067] The server converts the response message into audio data using a text-to-speech engine, and encodes and compresses it.

[0068] 8. Sending a Response (Server)

[0069] The generated voice data is transmitted to the terminal via the network.

[0070] 9. Response playback (terminal)

[0071] The device decodes and decompresses the received audio data and plays it through the speaker.

[0072] Specific examples

[0073] The operation of this system will be explained below using specific scenarios.

[0074] Example: Getting weather information

[0075] In the morning, a user speaks to a smart speaker in the living room, saying, "Good morning, tell me today's weather."

[0076] 1. The device records the audio and converts it into digital data.

[0077] 2. Send the digital data to the server.

[0078] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[0079] 4. The server parses this text and realizes that you are requesting weather information.

[0080] 5. Call an external weather forecast API to get current weather information.

[0081] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[0082] 7. Convert the response into audio data, encode it, and send it to the device.

[0083] 8. The device plays the audio data and tells the user the weather information.

[0084] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[0085] The processing flow will be explained below.

[0086] Step 1:

[0087] Receiving voice input (user, device)

[0088] The user speaks to the device, saying, "Tell me today's weather." The device receives the user's voice through a microphone and converts the voice data into digital data.

[0089] Step 2:

[0090] Audio data buffering (terminal)

[0091] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[0092] Step 3:

[0093] Audio data compression and encoding (terminal)

[0094] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[0095] Step 4:

[0096] Sending audio data (terminal)

[0097] The device sends the compressed and encoded audio data to the server via the network.

[0098] Step 5:

[0099] Receiving an HTTP request (server)

[0100] The server receives the HTTP request sent from the terminal and acquires the audio data.

[0101] Step 6:

[0102] Decoding and decompression (server)

[0103] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[0104] Step 7:

[0105] Speech recognition (server)

[0106] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[0107] Step 8:

[0108] Text analysis and natural language processing (server)

[0109] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[0110] Step 9:

[0111] Information acquisition (server)

[0112] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[0113] Step 10:

[0114] Generate a response message (server)

[0115] The server generates a natural response message based on the weather information it retrieves, for example, "It's sunny today and the temperature is 25 degrees."

[0116] Step 11:

[0117] Text-to-speech conversion (server)

[0118] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[0119] Step 12:

[0120] Audio data encoding and compression (server)

[0121] The server compresses and encodes the generated audio data into an efficient transmission format.

[0122] Step 13:

[0123] Sending a response (server)

[0124] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[0125] Step 14:

[0126] Decoding and decompression (terminal)

[0127] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[0128] Step 15:

[0129] Audio playback (device)

[0130] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "Today is sunny and the temperature is 25 degrees."

[0131] Example 1

[0132] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0133] Conventional voice assistant systems have difficulty accurately understanding the user's intent due to the low accuracy of speech recognition and natural language processing. They also suffer from poor response generation quality and insufficient information provided to the user. Furthermore, communication delays and the time required for voice data compression and encoding often impair real-time performance.

[0134] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0135] In this invention, the server includes means including a voice recognition engine that converts voice data into text data, natural language processing means that analyzes the text data and understands the user's intention, means for acquiring information from an external service based on the user's intention, means including a generative AI model that generates a response message in natural language from the acquired information, and means for transmitting the voice data to the terminal, thereby enabling a quick and accurate response to the user's voice command to be provided.

[0136] "Voice input" refers to voice commands or questions spoken by the user into a microphone.

[0137] "Converting to digital data" refers to the process of converting analog audio signals into digital format data.

[0138] "Compression and encoding" refers to the process of reducing the size of digital data and converting it into a particular format.

[0139] "Server" means a computer system that receives, processes, and transmits audio data over a network.

[0140] A "voice recognition engine" refers to software or a system for converting voice data into text data.

[0141] "Natural language processing means" refers to software or a system that analyzes text data and understands the user's intent.

[0142] A "generative AI model" refers to a machine learning model for generating natural language response messages.

[0143] "Text-to-speech engine" refers to software or a system for converting text data into speech data.

[0144] "Response message" refers to a natural language message generated in response to a user's voice command.

[0145] "Means of obtaining information from external services" refers to the systems or means for obtaining the necessary information from external data sources or APIs.

[0146] "Means for playing audio data" refers to a system or device for decoding text-to-speech data and providing it to a user as audio.

[0147] "Terminal" refers to the device through which the user inputs voice and receives and plays responses from the server.

[0148] MODE FOR CARRYING OUT THE INVENTION

[0149] System Overview

[0150] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[0151] System configuration

[0152] This system mainly consists of a terminal and a server.

[0153] Terminal

[0154] The device receives voice input from the user and records, converts, compresses, and encodes the voice. This includes a microphone, a processor, and network communication functions. The device transmits the voice data to a server and receives and plays back response voice data from the server. Specific devices include smart speakers and smartphones.

[0155] server

[0156] The server includes a speech recognition engine that receives voice data sent from the device and converts it into text data. It also has natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains the necessary information from external services and generates a natural language response message. The response message is converted into voice data using a text-to-speech engine and sent to the device. Specific technologies used include Google Cloud Speech-to-Text API, Hugging Face Transformers, OpenWeatherMap API, and Amazon Polly.

[0157] System Operation

[0158] 1. Receiving voice input (terminal)

[0159] The user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[0160] 2. Sending audio data (terminal)

[0161] The digital data is transmitted to a server over a network, typically a Wi-Fi network.

[0162] 3. Speech recognition (server)

[0163] The server decodes and decompresses the received audio data and converts it into text using a speech recognition engine, using the Google Cloud Speech-to-Text API.

[0164] 4. Natural Language Processing (Server)

[0165] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, getting weather information). The analysis is performed using Hugging Face Transformers.

[0166] 5. Information Acquisition (Server)

[0167] The server calls external services such as weather forecast APIs to obtain the latest weather information. In this case, we use the OpenWeatherMap API.

[0168] 6. Response Generation (Server)

[0169] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated. GPT-3 can be used as a generative AI model.

[0170] 7. Voice data generation (server)

[0171] The server converts the response message into audio data using a text-to-speech engine, encodes and compresses it, and generates the audio data using Amazon Polly.

[0172] 8. Sending a Response (Server)

[0173] The generated audio data is sent to the device via the network, usually using the HTTP protocol.

[0174] 9. Response playback (terminal)

[0175] The device decodes and decompresses the received audio data and plays it through the speaker.

[0176] Specific examples

[0177] The operation of this system will be explained below using specific scenarios.

[0178] Example: Getting weather information

[0179] In the morning, a user sits in the living room and asks their smart speaker, "Good morning, what's the weather today?"

[0180] 1. The device records the audio and converts it into digital data.

[0181] 2. Send the digital data to the server.

[0182] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[0183] 4. The server parses this text and realizes that you are requesting weather information.

[0184] 5. Call an external weather forecast API to get current weather information.

[0185] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[0186] 7. Convert the response into audio data, encode it, and send it to the device.

[0187] 8. The device plays the audio data and tells the user the weather information.

[0188] Prompt example

[0189] Below are examples of prompt sentences to input to the generative AI model.

[0190] example:

[0191] Provides information related to the user's voice command "What's the weather today?"

[0192] Weather information retrieval prompt:

[0193] User's voice command: "What's the weather like today?"

[0194] Get the response from the weather forecast API and generate a response message in the following format.

[0195] Format: Today is {weather} and the temperature is {temperature} degrees.

[0196] For example: "It's sunny today and the temperature is 25 degrees."

[0197] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[0198] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0199] Step 1: Receiving Audio Input (Device)

[0200] A user issues a voice command to a device (such as a smart speaker). For example, if the user says, "Tell me the weather today," the device's microphone receives this voice. The input is the user's voice, and the output is an analog voice signal. This analog signal is converted into digital data by a processor.

[0201] Step 2: Convert and encode audio data (device)

[0202] The terminal converts the received analog voice signal into digital data. This digital data is then compressed and encoded. The input is an analog voice signal, and the output is compressed and encoded digital data. This process uses an audio codec (e.g., AAC or MP3).

[0203] Step 3: Sending audio data (terminal)

[0204] The device sends the compressed and encoded audio data to the server using a network communication function (such as Wi-Fi). The input is the compressed and encoded digital data, and the output is an HTTP request sent to the server.

[0205] Step 4: Receive and decode the audio data (server)

[0206] The server receives the audio data sent from the device. The received data is decoded and decompressed. The input is compressed and encoded digital data, and the output is decoded and decompressed digital audio data. A decoder (e.g., AAC decoder) is used for this process.

[0207] Step 5: Speech Recognition (Server)

[0208] The server converts the decoded and decompressed audio data into text data using a speech recognition engine (e.g., speech recognition API). The input is the decoded and decompressed digital audio data, and the output is text data such as "What's the weather like today?"

[0209] Step 6: Natural Language Processing (Server)

[0210] The server analyzes the text data obtained by speech recognition using a natural language processing engine (e.g., a natural language processing library) to understand the user's intent. The input is text data, and the output is the analyzed user's intent (e.g., getting weather information).

[0211] Step 7: Information Acquisition (Server)

[0212] The server retrieves the necessary information (e.g., weather information) from an external service based on the analyzed user intent. For example, it calls a weather forecast API to retrieve current weather information. The input is the user intent, and the output is the weather information retrieved from the external service (e.g., sunny, temperature 25 degrees).

[0213] Step 8: Response Generation (Server)

[0214] The server uses a generative AI model to generate a natural language response message based on the acquired information. For example, it generates a message such as "It's sunny today, and the temperature is 25 degrees." The input is weather information, and the output is the text data of the response message.

[0215] Step 9: Generate audio data (server)

[0216] The server converts the generated response message into audio data using a text-to-speech engine, and encodes and compresses it. The input is the text data of the response message, and the output is the compressed and encoded audio data.

[0217] Step 10: Sending the Response (Server)

[0218] The server sends the generated audio data to the device via the network. The input is the compressed and encoded audio data, and the output is the transmission of an HTTP response to the device.

[0219] Step 11: Receive and Play Response (Terminal)

[0220] The device receives the audio data sent from the server, decodes and decompresses it. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data. It then plays the audio data through the device's speaker and provides the user with a voice response saying, "Today is sunny and the temperature is 25 degrees."

[0221] (Application example 1)

[0222] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0223] Traditionally, customer service in physical stores has relied on a limited number of staff, making it difficult to provide fast and accurate information, especially during busy times. Furthermore, when customers wanted information about a product, they had to search for it themselves, which took time and effort. This could lead to a decline in customer satisfaction. Therefore, there is a demand for a system that allows customers to easily obtain product information through voice input.

[0224] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0225] In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intent, means for acquiring information from an external service based on the user's intent, means for generating a response message in natural language from the acquired information, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, and the system is used to provide product information in a physical store, means for acquiring information from a product database based on the user's intent, and means for generating and playing the acquired information as a response message. This enables customers to quickly and accurately obtain product information through voice input.

[0226] "Voice input" is a means by which a user provides instructions or information to a system using voice.

[0227] "Digital data" refers to data that has been converted from analog audio information into a digital format.

[0228] A "server" is a computer system that provides various services over a network.

[0229] A "voice recognition engine" is software that analyzes input voice data and converts it into text data.

[0230] "Natural language processing means" refers to techniques and methods for analyzing text data and understanding user intent.

[0231] "External services" are other systems or APIs that the server uses in response to user requests.

[0232] A "response message" is a text response generated based on the user's intent.

[0233] A "text-to-speech engine" is software for converting text data into audio data.

[0234] A "terminal" is a hardware device that is directly operated by a user and receives and plays audio.

[0235] A "product database" is a collection of data that collects and stores product information from physical stores.

[0236] A "physical store" is a store that sells products at a physical location.

[0237] "Customers" refer to typical consumers who use physical stores.

[0238] System Overview

[0239] This invention relates to a system that enables customers in a physical store to quickly and accurately obtain product information through voice input. The system is primarily composed of hardware and software for receiving voice input, converting it to digital data, transmitting it to a server, performing voice recognition, natural language processing, information acquisition, response generation, and voice output.

[0240] Hardware and software used

[0241] Terminal: Smartphone or smart device (including microphone, processor, and network communication function)

[0242] Server: Speech recognition engine, natural language processing engine, text-to-speech engine

[0243] External service: Product database access API

[0244] Operating procedure

[0245] Usage example

[0246] Consider a scenario in which a customer uses a smartphone in a physical store to ask about product information. The customer speaks into the smartphone's microphone, saying, "Tell me the price of this product."

[0247] 1. Receiving voice input (terminal): The user speaks into the smartphone's microphone. The terminal converts this voice into digital data, compresses it, and encodes it.

[0248] 2. Sending digital data (terminal): The terminal sends the digital data to the server via network communication.

[0249] 3. Speech Recognition (Server): The server decodes and decompresses the received digital data and converts it into text data using a speech recognition engine.

[0250] 4. Natural Language Processing (Server): The server analyzes the text data and recognizes the customer's request (in this case, obtaining the product price).

[0251] 5. Information acquisition (server): The server calls the API to access the product database and acquires the price information of the relevant product.

[0252] 6. Response generation (server): Based on the obtained price information, a response message such as "The price of this product is XX yen" is generated.

[0253] 7. Generate voice data (Server): The server converts the response message into voice data using a text-to-speech engine, and encodes and compresses it.

[0254] 8. Sending response (server): The server sends the generated voice data to the terminal via the network.

[0255] 9. Playback of response (terminal): The terminal decodes and decompresses the received voice data and plays it back through the speaker. The customer receives a voice response such as "The price of this product is XX yen."

[0256] Specific examples

[0257] Examples of voice commands include prompts such as:

[0258] "Please tell me the price of this item."

[0259] "Tell me what products you recommend."

[0260] "What are the features of this product?"

[0261] When a customer says to their smartphone, "Tell me the price of this item," the system receives the voice input and quickly provides the product price information via voice, allowing customers to get the information they need without any hassle while in the store.

[0262] This is a specific embodiment for carrying out the invention.

[0263] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0264] Step 1:

[0265] The user provides voice input

[0266] The user speaks into the smartphone's microphone, saying, "Tell me the price of this item." The smartphone receives the voice data. In this process, the user's voice is input as an analog signal.

[0267] Step 2:

[0268] Converting audio into digital data

[0269] The terminal converts the received voice into digital data, which includes the processes of voice sampling and quantization, and outputs the analog voice data as digital data.

[0270] Step 3:

[0271] Sending digital data to a server

[0272] The terminal compresses and encodes the converted digital data and sends it to the server via network communication. The input is the converted digital data and the output is the compressed data sent to the server.

[0273] Step 4:

[0274] Converting audio data into text data

[0275] The server decodes and decompresses the received data and converts the voice data into text data using a speech recognition engine. The input is compressed voice data and the output is text data.

[0276] Step 5:

[0277] Analyzing text data to understand user intent

[0278] The server analyzes the text data and uses a natural language processing engine to understand the user's intent. In this process, the input is the text data to be analyzed, and the output is the intent data, such as "I want to know the price of this product."

[0279] Step 6:

[0280] Get product information

[0281] Based on the user's intent, the server calls the API to access the product database and obtains the price information of the relevant product. The input is the understood user intent, and the output is the obtained price information.

[0282] Step 7:

[0283] Generate a response message

[0284] The server generates a response message in natural language based on the acquired product information. The input for this process is product price information, and the output is a response message saying, "The price of this product is XX yen."

[0285] Step 8:

[0286] Converting response messages into audio data

[0287] The server converts the generated response message into voice data using a text-to-speech engine. The input is the response message, and the output is the voice data.

[0288] Step 9:

[0289] Sending audio data to the device

[0290] The server compresses and encodes the converted audio data and sends it to the terminal via network communication. The input is the generated audio data, and the output is the compressed data sent to the terminal.

[0291] Step 10:

[0292] Playing audio data

[0293] The terminal decodes and decompresses the received audio data and plays it through a speaker. The input is compressed audio data and the output is audio information played back to the user.

[0294] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0295] System Overview

[0296] This invention relates to an AI butler system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses accordingly. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[0297] System configuration

[0298] This system mainly consists of a terminal and a server.

[0299] Terminal

[0300] The device receives voice input from the user and records, converts, compresses, and encodes it. It includes a microphone, a processor, and network communication functions. The device transmits voice data to the server and receives and plays response voice data from the server.

[0301] server

[0302] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from external services and generates a response message in natural language. It also has an emotion recognition engine that analyzes the user's emotions and adjusts the response based on the emotions.

[0303] System Operation

[0304] The operation of the system of the present invention will now be described in detail.

[0305] Brief Description

[0306] 1. Receiving voice input (terminal)

[0307] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[0308] 2. Sending audio data (terminal)

[0309] The digital data is transmitted over a network to a server.

[0310] 3. Speech recognition (server)

[0311] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine, which generates the text "What's the weather like today?"

[0312] 4. Emotion Recognition (Server)

[0313] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, detecting, for example, whether the user is tired or happy.

[0314] 5. Text Analysis and Natural Language Processing (Server)

[0315] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[0316] 6. Information Acquisition (Server)

[0317] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[0318] 7. Generating a Response Message (Server)

[0319] The server generates a natural-sounding response message based on the weather information it has acquired, e.g., "It's sunny today, and the temperature is 25 degrees." It also adjusts the message to an appropriate tone, taking into account the user's emotions.

[0320] 8. Text-to-speech conversion (server)

[0321] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[0322] 9. Sending audio data (server)

[0323] The server transmits the generated voice data to the terminal via the network.

[0324] 10. Response playback (terminal)

[0325] The device decodes and decompresses the received audio data and plays it through the speaker.

[0326] Specific examples

[0327] The operation of this system will be explained below using specific scenarios.

[0328] Example: Obtaining weather information based on emotions

[0329] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[0330] 1. The device records the audio and converts it into digital data.

[0331] 2. Send the digital data to the server.

[0332] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[0333] 4. The server uses an emotion recognition engine to determine that the user is tired.

[0334] 5. The server understands that you are requesting weather information and retrieves the latest weather information from the weather forecast API.

[0335] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[0336] 7. The response message is converted into voice data and sent to the terminal.

[0337] 8. The device plays the audio data and tells the user, "It's sunny today and the temperature is 25 degrees. Good luck."

[0338] In this way, the system can take into account the user's emotions and provide more personalized responses, creating a more natural and comfortable interface for users, enabling them to efficiently search for complex information and perform tasks.

[0339] The processing flow will be explained below.

[0340] Step 1:

[0341] Receiving voice input (user, device)

[0342] The user speaks to the device, saying, "Tell me the weather today." The device's microphone converts this voice into digital data, capturing the voice data.

[0343] Step 2:

[0344] Audio data buffering (terminal)

[0345] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[0346] Step 3:

[0347] Audio data compression and encoding (terminal)

[0348] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[0349] Step 4:

[0350] Sending audio data (terminal)

[0351] The device sends the compressed and encoded audio data to the server via the network.

[0352] Step 5:

[0353] Receiving an HTTP request (server)

[0354] The server receives the HTTP request sent from the terminal and acquires the audio data.

[0355] Step 6:

[0356] Decoding and decompression (server)

[0357] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[0358] Step 7:

[0359] Speech recognition (server)

[0360] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[0361] Step 8:

[0362] Emotion recognition (server)

[0363] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, for example, detecting whether the user is tired or happy from the tone of their voice and speaking patterns.

[0364] Step 9:

[0365] Text analysis and natural language processing (server)

[0366] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[0367] Step 10:

[0368] Information acquisition (server)

[0369] In response to a user request, the server calls the weather forecast API of an external service to obtain current weather information.

[0370] Step 11:

[0371] Generate a response message (server)

[0372] The server generates a natural-sounding response message based on the acquired weather information. For example, the generated message might be, "It's sunny today, and the temperature is 25 degrees." In addition, it adds encouraging words such as "Keep up the good work!" based on the user's emotions (e.g., tiredness) analyzed by an emotion recognition engine.

[0373] Step 12:

[0374] Text-to-speech conversion (server)

[0375] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[0376] Step 13:

[0377] Audio data encoding and compression (server)

[0378] The server compresses and encodes the generated audio data into an efficient transmission format.

[0379] Step 14:

[0380] Sending a response (server)

[0381] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[0382] Step 15:

[0383] Decoding and decompression (terminal)

[0384] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[0385] Step 16:

[0386] Audio playback (device)

[0387] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "It's sunny today, and the temperature is 25 degrees. Good luck!"

[0388] Example 2

[0389] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] Conventional speech recognition systems simply convert a user's speech into text and understand their intent, but it is difficult to generate responses that take the user's emotions into account. As a result, many responses are mechanical and ignore the user's emotional state, resulting in a limited user experience. Furthermore, the entire process from voice input to response generation requires immediacy and accuracy of information, but there has been a lack of technology to achieve this.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0392] In this invention, the server includes means including a voice recognition engine for converting voice data into text data, natural language processing and emotion recognition means for analyzing the text data and understanding the user's intentions and emotions, and means for acquiring information from an external database, thereby enabling the server to provide a more personalized response to the user by converting the user's voice input into digital data and analyzing the user's emotions as well as their intentions to generate an appropriate response message.

[0393] "Voice input" refers to an input format in which a user gives instructions or asks questions using voice.

[0394] "Digital data" refers to data obtained by converting analog audio signals into digital format.

[0395] A "server" is a computer system that communicates with other computers and devices over a network and provides specific functions or services.

[0396] A "speech recognition engine" is a piece of software or hardware that converts voice data into text data.

[0397] "Natural language processing" is a technology that enables computers to understand, analyze, and generate human language.

[0398] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data and text data.

[0399] An "external database" is a collection of information that exists outside the system and is a database that is accessed to obtain required information.

[0400] "Text data" is character string data converted from voice input by a voice recognition engine.

[0401] A "response message" is a reply in natural language format that the system generates in response to a user's question or instruction.

[0402] A "text-to-speech engine" is a piece of software or hardware for converting text data into speech data.

[0403] A "terminal" is a device that allows a user to input voice and has the function of communicating with a server.

[0404] "Decoding" is the process of returning encoded data to its original form.

[0405] "Decompression" is the process of restoring compressed data to its original state.

[0406] System Overview

[0407] This invention relates to an AI system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses based on those emotions. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[0408] System configuration

[0409] This system mainly consists of a terminal and a server.

[0410] Terminal

[0411] The device receives voice input from the user and records, converts, compresses, and encodes it. This includes a microphone, a processor, and network communication capabilities. Specifically, the device's microphone (e.g., a microphone) converts the voice into digital data, and the processor (e.g., a central processing unit) compresses and encodes the data. The device then uses its network communication capabilities (e.g., a wireless communication module) to send the data to a server, receive response voice data from the server, and play it back.

[0412] server

[0413] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. For example, the server converts the received voice data into text data using a speech recognition engine (e.g., generic voice recognition software). The server also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server retrieves necessary information from an external database and generates a natural language response message. The server also has an emotion recognition engine (e.g., generic emotion analysis software) that adjusts the response based on the user's emotion.

[0414] System Operation

[0415] Receiving and processing audio input (terminal)

[0416] When a user speaks a voice command into the device, such as "What's the weather today?", the device's microphone converts the voice into digital data, which is then compressed and encoded by a processor and sent to a server via network communications.

[0417] Speech and emotion recognition (server)

[0418] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine. For example, text data such as "What's the weather like today?" is generated. Then, an emotion recognition engine is used to analyze the user's emotions from the voice data. For example, emotions such as tiredness or happiness can be detected.

[0419] Text analysis and information retrieval (server)

[0420] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. For example, it determines that the user is "requesting weather information." The server then calls an external database (e.g., a general-name weather forecast database) to obtain current weather information.

[0421] Generate and send a response message (server)

[0422] The server generates a natural-sounding response message based on the acquired weather information. For example, it might generate a message like "It's sunny today, and the temperature is 25 degrees." Taking the user's emotions into consideration, the server might add words of encouragement, such as "Good luck." The server then converts the generated text response message into audio data using a text-to-speech engine and sends it to the device.

[0423] Response playback (terminal)

[0424] The device decodes and decompresses the received audio data and plays it back through a speaker (e.g., a general-purpose audio output device). Specifically, it conveys to the user something like, "Today is sunny, and the temperature is 25 degrees. Good luck."

[0425] Specific examples

[0426] The operation of this system will be explained below using specific scenarios.

[0427] Example: Obtaining weather information based on emotions

[0428] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[0429] 1. The device records the sound and converts it into digital data (microphone: the common name for a microphone).

[0430] 2. Send the digital data to the server (network communication: commonly known as a wireless communication module).

[0431] 3. The server converts the voice data into text (voice recognition engine: generic name for voice recognition software) and recognizes the text as "Tell me what the weather is today."

[0432] 4. The server uses an emotion recognition engine (commonly known as emotion analysis software) to determine that the user is tired.

[0433] 5. The server understands that you are requesting weather information (natural language processing engine: a general term for natural language processing software) and retrieves the latest weather information from a weather forecast database (a general term for an external database).

[0434] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[0435] 7. The response message is converted into voice data (text-to-speech engine: a common name for speech synthesis software) and sent to the terminal.

[0436] 8. The device plays audio data (speaker: a common name for an audio output device) and tells the user, "It's sunny today, and the temperature is 25 degrees. Good luck."

[0437] Prompt Sentence Examples

[0438] "Tell me about today's weather."

[0439] "Check tomorrow's schedule."

[0440] "Play some uplifting music"

[0441] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0442] Step 1:

[0443] The device receives voice input from the user. When the user says to the device, "What's the weather today?", the microphone converts the voice into digital data. The input is analog voice and the output is digital data. This data is then processed by a processor within the device.

[0444] Step 2:

[0445] The device compresses and encodes the digital data. Specifically, a processor (e.g., a central processing unit) compresses the digital data into MP3 or AAC format and encodes the data into a format suitable for the network. The input is digital audio data, and the output is compressed and encoded audio data.

[0446] Step 3:

[0447] The device sends the compressed and encoded audio data to the server via a network communication function (e.g., a wireless communication module). The data is encrypted to ensure secure communication. The input is the compressed and encoded audio data, and the output is the transmission of the audio data to the server.

[0448] Step 4:

[0449] The server decodes and decompresses the received audio data, specifically using the server's (e.g., computer system's) decoding and decompression capabilities (e.g., generic software) to return the audio data to its original digital form. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data.

[0450] Step 5:

[0451] The server uses a speech recognition engine (e.g., speech recognition software) to convert the decoded and decompressed audio data into text data. Specifically, the speech recognition engine analyzes the audio data and generates the text "What's the weather like today?" The input is the decoded audio data, and the output is text data.

[0452] Step 6:

[0453] The server uses an emotion recognition engine (e.g., emotion analysis software) to analyze the user's emotions from the voice data. Specifically, the emotion recognition engine analyzes the tone and pitch of the voice to identify the user's emotions, such as tiredness or joy. The input is the voice data, and the output is the user's emotional information as a result of the analysis.

[0454] Step 7:

[0455] The server uses a natural language processing engine (e.g., natural language processing software) to analyze the text data and understand the user's intent. Specifically, the natural language processing engine processes the text data and determines that it is a "request for weather information." The input is the text data and the user's emotional information, and the output is the analysis result of the user's intent.

[0456] Step 8:

[0457] The server, according to the user's request, calls an external database (e.g., a weather forecast database) to obtain current weather information. Specifically, the server sends a request to the database via an API to obtain weather information. The input is the user's intent and the API request from the external database, and the output is weather information.

[0458] Step 9:

[0459] The server generates a natural response message based on the acquired weather information. Specifically, the server uses its response message generation function to generate a message such as "It's sunny today, and the temperature is 25 degrees. Good luck!" The input is weather information and the user's emotional information, and the output is the response message.

[0460] Step 10:

[0461] The server uses a text-to-speech engine (e.g., speech synthesis software) to convert the generated text response message into audio data. Specifically, the text-to-speech engine converts the text data into audio data. The input is the response message, and the output is audio data.

[0462] Step 11:

[0463] The server sends the generated voice data to the device via the network communication function. The data is again encrypted to ensure secure communication. The input is voice data, and the output is the transmission of voice data to the device.

[0464] Step 12:

[0465] The device decodes and decompresses the received voice data and plays it back through the speaker. Specifically, the device uses its decoding function to return the voice data to its original format and plays back from the speaker, "It's sunny today, and the temperature is 25 degrees. Good luck." The input is the voice data received from the server, and the output is a voice response to the user.

[0466] (Application example 2)

[0467] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0468] To improve passenger comfort in autonomous vehicles, systems that can accurately understand passenger emotions and intentions and provide appropriate responses based on those emotions are required. However, current voice assistant systems for autonomous vehicles lack emotion recognition capabilities and are unable to provide personalized responses based on passenger emotions, which can cause discomfort to passengers.

[0469] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intention, means for acquiring information from an external service based on the user's intention, means for generating the acquired information into a response message in natural language, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, means for analyzing the user's emotion from the voice data, and means for adjusting the response message based on the user's emotion. This enables passengers in an autonomous vehicle to receive a personalized response based on their emotions.

[0470] The "means for receiving user voice input" is a device or function that receives voice commands from the user through a microphone.

[0471] The "means for converting the voice input into digital data" refers to a device or function that converts analog input voice data into digital data.

[0472] The "means for transmitting the digital data to the server" refers to a device or function that transmits the digital data to the server via network communication.

[0473] The "means including a voice recognition engine for converting said digital data into text data" refers to a device or function that converts digital voice data into text form using voice recognition technology.

[0474] The "natural language processing means for analyzing the text data and understanding the user's intent" is a device or function that uses natural language processing technology to analyze the text data and understand the user's intent.

[0475] The "means for acquiring information from an external service based on the user's intention" is a device or function for acquiring necessary information from an external information providing service based on the analyzed user's intention.

[0476] The "means for generating a response message in a natural language from the acquired information" refers to a device or function that generates a response message in a natural language based on the acquired information.

[0477] The "means including a text-to-speech engine for converting the response message into voice data" refers to a device or function that converts the generated text message into voice data.

[0478] The "means for transmitting the voice data to the terminal" is a device or function for transmitting the generated voice data to the terminal.

[0479] The "means for reproducing the audio data" refers to a device or function such as a speaker that reproduces audio data on the terminal.

[0480] The "means for analyzing the user's emotions from the voice data" refers to a device or function that analyzes the voice data and identifies the user's emotional state.

[0481] The "means for adjusting the response message based on the user's emotions" refers to a device or function that adjusts the content and tone of the response message based on the analyzed emotional state of the user.

[0482] A "means for receiving voice input from a passenger of an automated vehicle" is a device or function within the automated vehicle that receives voice input from a passenger.

[0483] A "generative AI model" refers to a trained artificial intelligence model used for natural language processing and intent analysis.

[0484] A "prompt sentence" is an input sentence containing instructions or questions from a user given to a generative AI model.

[0485]

[0486] This invention relates to a voice assistant system for improving passenger comfort in autonomous vehicles. The system can provide personalized services to passengers by using AI technology to receive user voice input, analyze it, and generate appropriate responses.

[0487] System Configuration

[0488] This system mainly consists of a terminal and a server.

[0489] Terminal

[0490] The terminal is installed inside the autonomous vehicle and has the following functions:

[0491] 1. Audio input reception:

[0492] Passenger voice commands are received via a microphone.

[0493] 2. Audio digital conversion:

[0494] Converts audio input into digital data.

[0495] 3. Data transmission:

[0496] The converted digital data is sent to a server.

[0497] 4. Audio data playback:

[0498] The response voice data sent from the server is played back.

[0499] server

[0500] The server receives the digital audio data sent from the terminal and provides the following functions:

[0501] 1. Speech Recognition Engine:

[0502] Converts digital voice data into text data using the Google Cloud Speech-to-Text API or similar.

[0503] 2. Natural Language Processing Engine:

[0504] The converted text data is analyzed to understand the user's intent. OpenAI's generative AI models (such as GPT-3 / 4) are used.

[0505] 3. Emotion Recognition Engine:

[0506] Analyzes user emotions from voice data, using Azure Emotion API etc.

[0507] 4. Information acquisition means:

[0508] Obtain necessary information from external services based on the user's intent.

[0509] 5. Response message generation:

[0510] A natural language response message is generated based on the acquired information and the results of sentiment analysis.

[0511] 6. Text-to-speech engine:

[0512] The generated text message is converted into audio data, using the Google Text-to-Speech API or similar.

[0513] 7. Means of data transmission:

[0514] The converted audio data is sent to the terminal.

[0515] System Operation

[0516] The system works as follows:

[0517] 1. Receiving voice input:

[0518] The user (passenger) issues a voice command to the terminal, for example, a prompt such as "What's the weather like today?"

[0519] 2. Data transmission:

[0520] The terminal transmits the voice data to the server.

[0521] 3. Speech Recognition and Natural Language Processing:

[0522] The server converts the received voice data into text and analyzes it using a natural language processing engine to understand the user's intent.

[0523] 4. Emotion recognition:

[0524] An emotion recognition engine is used to analyze the user's emotions and tailor the response message based on the emotions.

[0525] 5. Information Acquisition and Response Generation:

[0526] External information is acquired based on the user's intention, and a personalized response message is generated.

[0527] 6. Audio data conversion and transmission:

[0528] The text-to-speech engine converts the response message into voice data, which is then sent to the terminal.

[0529] 7. Response playback:

[0530] The terminal plays back the transmitted voice data and conveys the response to the passenger.

[0531] Specific examples

[0532] For example, if a passenger asks the terminal, "What's the weather like today?", and the passenger sounds a little tired, the system works as follows:

[0533] 1. The device receives the audio, converts it into digital data, and sends it to the server.

[0534] 2. The server uses a speech recognition engine to convert the text into "What's the weather like today?"

[0535] 3. Use an emotion recognition engine to detect tired emotions from voice data.

[0536] 4. The natural language processing engine analyzes the user's intent and understands that they are looking for weather information.

[0537] 5. The server retrieves the latest weather information from the weather forecast API.

[0538] 6. Based on the emotion recognition results, generate a response message such as "It's a sunny day today. Please relax."

[0539] 7. The response message is converted into voice data and sent to the terminal.

[0540] 8. The device plays audio data telling the passenger, "It's a sunny day today. Please relax."

[0541] An example prompt is:

[0542] "Please tell me the weather today."

[0543] "Please tell me the current traffic information."

[0544] This enables the system to provide passengers with personalized services and a more comfortable travel experience.

[0545] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0546] Step 1: Receiving audio input

[0547] The terminal receives voice commands from the user (passenger) using a microphone. The input is the passenger's voice, and the output is analog voice data. Specifically, the microphone captures the voice and converts it into a digital signal.

[0548] Step 2: Sending audio data

[0549] The terminal transmits the converted digital audio data to the server. The input is the digital audio data, and the output is the data transmitted to the server. Specifically, the terminal transmits the data to the server via network communication.

[0550] Step 3: Voice Recognition

[0551] The server converts the received digital voice data into text data using a speech recognition engine. The input is digital voice data, and the output is text data. Specifically, the server calls the Google Cloud Speech-to-Text API to obtain the text data.

[0552] Step 4: Natural Language Processing

[0553] The server analyzes the text data using a natural language processing engine to understand the user's intent. The input is text data, and the output is the analyzed intent. Specifically, the server uses OpenAI's generative AI model (GPT-3 / 4) to analyze the text data and determine the intent.

[0554] Step 5: Emotion Recognition

[0555] The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The input is the voice data, and the output is the user's emotional state. Specifically, the server calls the Azure Emotion API to analyze the voice data and identify the emotion.

[0556] Step 6: Information Acquisition

[0557] The server retrieves the necessary information from external services based on the analyzed intent. The input is the user's intent, and the output is the retrieved information. Specifically, the server retrieves information by calling external weather forecast APIs and traffic information APIs.

[0558] Step 7: Generate a response message

[0559] The server generates a response message in natural language based on the acquired information and the results of emotion analysis. The input is the acquired information and the user's emotional state, and the output is the response message. Specifically, the server uses the generative AI model to create a text message.

[0560] Step 8: Text-to-speech

[0561] The server converts the generated text message into audio data. The input is the response message, and the output is audio data. Specifically, the server calls the Google Text-to-Speech API to convert the text into audio data.

[0562] Step 9: Sending audio data

[0563] The server transmits the converted voice data to the terminal. The input is the voice data, and the output is the data transmitted to the terminal. Specifically, the server transmits the voice data to the terminal via network communication.

[0564] Step 10: Playing back the response

[0565] The terminal plays the received voice data. The input is voice data, and the output is the voice heard by the passenger. Specifically, the voice data is played through the terminal's speaker to convey the response to the passenger.

[0566] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0567] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0568] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0569] [Second embodiment]

[0570] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0571] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0572] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0573] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0574] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0575] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0576] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0577] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0578] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0579] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0580] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0581] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0582] System Overview

[0583] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[0584] System configuration

[0585] This system mainly consists of a terminal and a server.

[0586] Terminal

[0587] The terminal receives voice input from the user and records, converts, compresses, and encodes the voice. It includes a microphone, a processor, and network communication functions. The terminal transmits the voice data to the server and receives and plays back response voice data from the server.

[0588] server

[0589] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from an external service and generates a response message in natural language. The response message is converted into voice data using a text-to-speech engine and sent to the terminal.

[0590] System Operation

[0591] The operation of the system of the present invention will now be described in detail.

[0592] Brief Description

[0593] 1. Receiving voice input (terminal)

[0594] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[0595] 2. Sending audio data (terminal)

[0596] The digital data is transmitted over a network to a server.

[0597] 3. Speech recognition (server)

[0598] The server decodes and decompresses the received voice data and converts it into text data using a voice recognition engine.

[0599] 4. Natural Language Processing (Server)

[0600] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, obtaining weather information).

[0601] 5. Information Acquisition (Server)

[0602] The server calls external services such as weather forecast APIs to obtain the latest weather information.

[0603] 6. Response Generation (Server)

[0604] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated.

[0605] 7. Voice data generation (server)

[0606] The server converts the response message into audio data using a text-to-speech engine, and encodes and compresses it.

[0607] 8. Sending a Response (Server)

[0608] The generated voice data is transmitted to the terminal via the network.

[0609] 9. Response playback (terminal)

[0610] The device decodes and decompresses the received audio data and plays it through the speaker.

[0611] Specific examples

[0612] The operation of this system will be explained below using specific scenarios.

[0613] Example: Getting weather information

[0614] In the morning, a user speaks to a smart speaker in the living room, saying, "Good morning, tell me today's weather."

[0615] 1. The device records the audio and converts it into digital data.

[0616] 2. Send the digital data to the server.

[0617] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[0618] 4. The server parses this text and realizes that you are requesting weather information.

[0619] 5. Call an external weather forecast API to get current weather information.

[0620] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[0621] 7. Convert the response into audio data, encode it, and send it to the device.

[0622] 8. The device plays the audio data and tells the user the weather information.

[0623] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[0624] The processing flow will be explained below.

[0625] Step 1:

[0626] Receiving voice input (user, device)

[0627] The user speaks to the device, saying, "Tell me today's weather." The device receives the user's voice through a microphone and converts the voice data into digital data.

[0628] Step 2:

[0629] Audio data buffering (terminal)

[0630] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[0631] Step 3:

[0632] Audio data compression and encoding (terminal)

[0633] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[0634] Step 4:

[0635] Sending audio data (terminal)

[0636] The device sends the compressed and encoded audio data to the server via the network.

[0637] Step 5:

[0638] Receiving an HTTP request (server)

[0639] The server receives the HTTP request sent from the terminal and acquires the audio data.

[0640] Step 6:

[0641] Decoding and decompression (server)

[0642] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[0643] Step 7:

[0644] Speech recognition (server)

[0645] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[0646] Step 8:

[0647] Text analysis and natural language processing (server)

[0648] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[0649] Step 9:

[0650] Information acquisition (server)

[0651] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[0652] Step 10:

[0653] Generate a response message (server)

[0654] The server generates a natural response message based on the weather information it retrieves, for example, "It's sunny today and the temperature is 25 degrees."

[0655] Step 11:

[0656] Text-to-speech conversion (server)

[0657] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[0658] Step 12:

[0659] Audio data encoding and compression (server)

[0660] The server compresses and encodes the generated audio data into an efficient transmission format.

[0661] Step 13:

[0662] Sending a response (server)

[0663] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[0664] Step 14:

[0665] Decoding and decompression (terminal)

[0666] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[0667] Step 15:

[0668] Audio playback (device)

[0669] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "Today is sunny and the temperature is 25 degrees."

[0670] Example 1

[0671] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0672] Conventional voice assistant systems have difficulty accurately understanding the user's intent due to the low accuracy of speech recognition and natural language processing. They also suffer from poor response generation quality and insufficient information provided to the user. Furthermore, communication delays and the time required for voice data compression and encoding often impair real-time performance.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0674] In this invention, the server includes means including a voice recognition engine that converts voice data into text data, natural language processing means that analyzes the text data and understands the user's intention, means for acquiring information from an external service based on the user's intention, means including a generative AI model that generates a response message in natural language from the acquired information, and means for transmitting the voice data to the terminal, thereby enabling a quick and accurate response to the user's voice command to be provided.

[0675] "Voice input" refers to voice commands or questions spoken by the user into a microphone.

[0676] "Converting to digital data" refers to the process of converting analog audio signals into digital format data.

[0677] "Compression and encoding" refers to the process of reducing the size of digital data and converting it into a particular format.

[0678] "Server" means a computer system that receives, processes, and transmits audio data over a network.

[0679] A "voice recognition engine" refers to software or a system for converting voice data into text data.

[0680] "Natural language processing means" refers to software or a system that analyzes text data and understands the user's intent.

[0681] A "generative AI model" refers to a machine learning model for generating natural language response messages.

[0682] "Text-to-speech engine" refers to software or a system for converting text data into speech data.

[0683] "Response message" refers to a natural language message generated in response to a user's voice command.

[0684] "Means of obtaining information from external services" refers to the systems or means for obtaining the necessary information from external data sources or APIs.

[0685] "Means for playing audio data" refers to a system or device for decoding text-to-speech data and providing it to a user as audio.

[0686] "Terminal" refers to the device through which the user inputs voice and receives and plays responses from the server.

[0687] MODE FOR CARRYING OUT THE INVENTION

[0688] System Overview

[0689] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[0690] System configuration

[0691] This system mainly consists of a terminal and a server.

[0692] Terminal

[0693] The device receives voice input from the user and records, converts, compresses, and encodes the voice. This includes a microphone, a processor, and network communication functions. The device transmits the voice data to a server and receives and plays back response voice data from the server. Specific devices include smart speakers and smartphones.

[0694] server

[0695] The server includes a speech recognition engine that receives voice data sent from the device and converts it into text data. It also has natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains the necessary information from external services and generates a natural language response message. The response message is converted into voice data using a text-to-speech engine and sent to the device. Specific technologies used include Google Cloud Speech-to-Text API, Hugging Face Transformers, OpenWeatherMap API, and Amazon Polly.

[0696] System Operation

[0697] 1. Receiving voice input (terminal)

[0698] The user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[0699] 2. Sending audio data (terminal)

[0700] The digital data is transmitted to a server over a network, typically a Wi-Fi network.

[0701] 3. Speech recognition (server)

[0702] The server decodes and decompresses the received audio data and converts it into text using a speech recognition engine, using the Google Cloud Speech-to-Text API.

[0703] 4. Natural Language Processing (Server)

[0704] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, getting weather information). The analysis is performed using Hugging Face Transformers.

[0705] 5. Information Acquisition (Server)

[0706] The server calls external services such as weather forecast APIs to obtain the latest weather information. In this case, we use the OpenWeatherMap API.

[0707] 6. Response Generation (Server)

[0708] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated. GPT-3 can be used as a generative AI model.

[0709] 7. Voice data generation (server)

[0710] The server converts the response message into audio data using a text-to-speech engine, encodes and compresses it, and generates the audio data using Amazon Polly.

[0711] 8. Sending a Response (Server)

[0712] The generated audio data is sent to the device via the network, usually using the HTTP protocol.

[0713] 9. Response playback (terminal)

[0714] The device decodes and decompresses the received audio data and plays it through the speaker.

[0715] Specific examples

[0716] The operation of this system will be explained below using specific scenarios.

[0717] Example: Getting weather information

[0718] In the morning, a user sits in the living room and asks their smart speaker, "Good morning, what's the weather today?"

[0719] 1. The device records the audio and converts it into digital data.

[0720] 2. Send the digital data to the server.

[0721] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[0722] 4. The server parses this text and realizes that you are requesting weather information.

[0723] 5. Call an external weather forecast API to get current weather information.

[0724] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[0725] 7. Convert the response into audio data, encode it, and send it to the device.

[0726] 8. The device plays the audio data and tells the user the weather information.

[0727] Prompt example

[0728] Below are examples of prompt sentences to input to the generative AI model.

[0729] example:

[0730] Provides information related to the user's voice command "What's the weather today?"

[0731] Weather information retrieval prompt:

[0732] User's voice command: "What's the weather like today?"

[0733] Get the response from the weather forecast API and generate a response message in the following format.

[0734] Format: Today is {weather} and the temperature is {temperature} degrees.

[0735] For example: "It's sunny today and the temperature is 25 degrees."

[0736] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[0737] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0738] Step 1: Receiving Audio Input (Device)

[0739] A user issues a voice command to a device (such as a smart speaker). For example, if the user says, "Tell me the weather today," the device's microphone receives this voice. The input is the user's voice, and the output is an analog voice signal. This analog signal is converted into digital data by a processor.

[0740] Step 2: Convert and encode audio data (device)

[0741] The terminal converts the received analog voice signal into digital data. This digital data is then compressed and encoded. The input is an analog voice signal, and the output is compressed and encoded digital data. This process uses an audio codec (e.g., AAC or MP3).

[0742] Step 3: Sending audio data (terminal)

[0743] The device sends the compressed and encoded audio data to the server using a network communication function (such as Wi-Fi). The input is the compressed and encoded digital data, and the output is an HTTP request sent to the server.

[0744] Step 4: Receive and decode the audio data (server)

[0745] The server receives the audio data sent from the device. The received data is decoded and decompressed. The input is compressed and encoded digital data, and the output is decoded and decompressed digital audio data. A decoder (e.g., AAC decoder) is used for this process.

[0746] Step 5: Speech Recognition (Server)

[0747] The server converts the decoded and decompressed audio data into text data using a speech recognition engine (e.g., speech recognition API). The input is the decoded and decompressed digital audio data, and the output is text data such as "What's the weather like today?"

[0748] Step 6: Natural Language Processing (Server)

[0749] The server analyzes the text data obtained by speech recognition using a natural language processing engine (e.g., a natural language processing library) to understand the user's intent. The input is text data, and the output is the analyzed user's intent (e.g., getting weather information).

[0750] Step 7: Information Acquisition (Server)

[0751] The server retrieves the necessary information (e.g., weather information) from an external service based on the analyzed user intent. For example, it calls a weather forecast API to retrieve current weather information. The input is the user intent, and the output is the weather information retrieved from the external service (e.g., sunny, temperature 25 degrees).

[0752] Step 8: Response Generation (Server)

[0753] The server uses a generative AI model to generate a natural language response message based on the acquired information. For example, it generates a message such as "It's sunny today, and the temperature is 25 degrees." The input is weather information, and the output is the text data of the response message.

[0754] Step 9: Generate audio data (server)

[0755] The server converts the generated response message into audio data using a text-to-speech engine, and encodes and compresses it. The input is the text data of the response message, and the output is the compressed and encoded audio data.

[0756] Step 10: Sending the Response (Server)

[0757] The server sends the generated audio data to the device via the network. The input is the compressed and encoded audio data, and the output is the transmission of an HTTP response to the device.

[0758] Step 11: Receive and Play Response (Terminal)

[0759] The device receives the audio data sent from the server, decodes and decompresses it. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data. It then plays the audio data through the device's speaker and provides the user with a voice response saying, "Today is sunny and the temperature is 25 degrees."

[0760] (Application example 1)

[0761] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0762] Traditionally, customer service in physical stores has relied on a limited number of staff, making it difficult to provide fast and accurate information, especially during busy times. Furthermore, when customers wanted information about a product, they had to search for it themselves, which took time and effort. This could lead to a decline in customer satisfaction. Therefore, there is a demand for a system that allows customers to easily obtain product information through voice input.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0764] In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intent, means for acquiring information from an external service based on the user's intent, means for generating a response message in natural language from the acquired information, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, and the system is used to provide product information in a physical store, means for acquiring information from a product database based on the user's intent, and means for generating and playing the acquired information as a response message. This enables customers to quickly and accurately obtain product information through voice input.

[0765] "Voice input" is a means by which a user provides instructions or information to a system using voice.

[0766] "Digital data" refers to data that has been converted from analog audio information into a digital format.

[0767] A "server" is a computer system that provides various services over a network.

[0768] A "voice recognition engine" is software that analyzes input voice data and converts it into text data.

[0769] "Natural language processing means" refers to techniques and methods for analyzing text data and understanding user intent.

[0770] "External services" are other systems or APIs that the server uses in response to user requests.

[0771] A "response message" is a text response generated based on the user's intent.

[0772] A "text-to-speech engine" is software for converting text data into audio data.

[0773] A "terminal" is a hardware device that is directly operated by a user and receives and plays audio.

[0774] A "product database" is a collection of data that collects and stores product information from physical stores.

[0775] A "physical store" is a store that sells products at a physical location.

[0776] "Customers" refer to typical consumers who use physical stores.

[0777] System Overview

[0778] This invention relates to a system that enables customers in a physical store to quickly and accurately obtain product information through voice input. The system is primarily composed of hardware and software for receiving voice input, converting it to digital data, transmitting it to a server, performing voice recognition, natural language processing, information acquisition, response generation, and voice output.

[0779] Hardware and software used

[0780] Terminal: Smartphone or smart device (including microphone, processor, and network communication function)

[0781] Server: Speech recognition engine, natural language processing engine, text-to-speech engine

[0782] External service: Product database access API

[0783] Operating procedure

[0784] Usage example

[0785] Consider a scenario in which a customer uses a smartphone in a physical store to ask about product information. The customer speaks into the smartphone's microphone, saying, "Tell me the price of this product."

[0786] 1. Receiving voice input (terminal): The user speaks into the smartphone's microphone. The terminal converts this voice into digital data, compresses it, and encodes it.

[0787] 2. Sending digital data (terminal): The terminal sends the digital data to the server via network communication.

[0788] 3. Speech Recognition (Server): The server decodes and decompresses the received digital data and converts it into text data using a speech recognition engine.

[0789] 4. Natural Language Processing (Server): The server analyzes the text data and recognizes the customer's request (in this case, obtaining the product price).

[0790] 5. Information acquisition (server): The server calls the API to access the product database and acquires the price information of the relevant product.

[0791] 6. Response generation (server): Based on the obtained price information, a response message such as "The price of this product is XX yen" is generated.

[0792] 7. Generate voice data (Server): The server converts the response message into voice data using a text-to-speech engine, and encodes and compresses it.

[0793] 8. Sending response (server): The server sends the generated voice data to the terminal via the network.

[0794] 9. Playback of response (terminal): The terminal decodes and decompresses the received voice data and plays it back through the speaker. The customer receives a voice response such as "The price of this product is XX yen."

[0795] Specific examples

[0796] Examples of voice commands include prompts such as:

[0797] "Please tell me the price of this item."

[0798] "Tell me what products you recommend."

[0799] "What are the features of this product?"

[0800] When a customer says to their smartphone, "Tell me the price of this item," the system receives the voice input and quickly provides the product price information via voice, allowing customers to get the information they need without any hassle while in the store.

[0801] This is a specific embodiment for carrying out the invention.

[0802] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0803] Step 1:

[0804] The user provides voice input

[0805] The user speaks into the smartphone's microphone, saying, "Tell me the price of this item." The smartphone receives the voice data. In this process, the user's voice is input as an analog signal.

[0806] Step 2:

[0807] Converting audio into digital data

[0808] The terminal converts the received voice into digital data, which includes the processes of voice sampling and quantization, and outputs the analog voice data as digital data.

[0809] Step 3:

[0810] Sending digital data to a server

[0811] The terminal compresses and encodes the converted digital data and sends it to the server via network communication. The input is the converted digital data and the output is the compressed data sent to the server.

[0812] Step 4:

[0813] Converting audio data into text data

[0814] The server decodes and decompresses the received data and converts the voice data into text data using a speech recognition engine. The input is compressed voice data and the output is text data.

[0815] Step 5:

[0816] Analyzing text data to understand user intent

[0817] The server analyzes the text data and uses a natural language processing engine to understand the user's intent. In this process, the input is the text data to be analyzed, and the output is the intent data, such as "I want to know the price of this product."

[0818] Step 6:

[0819] Get product information

[0820] Based on the user's intent, the server calls the API to access the product database and obtains the price information of the relevant product. The input is the understood user intent, and the output is the obtained price information.

[0821] Step 7:

[0822] Generate a response message

[0823] The server generates a response message in natural language based on the acquired product information. The input for this process is product price information, and the output is a response message saying, "The price of this product is XX yen."

[0824] Step 8:

[0825] Converting response messages into audio data

[0826] The server converts the generated response message into voice data using a text-to-speech engine. The input is the response message, and the output is the voice data.

[0827] Step 9:

[0828] Sending audio data to the device

[0829] The server compresses and encodes the converted audio data and sends it to the terminal via network communication. The input is the generated audio data, and the output is the compressed data sent to the terminal.

[0830] Step 10:

[0831] Playing audio data

[0832] The terminal decodes and decompresses the received audio data and plays it through a speaker. The input is compressed audio data and the output is audio information played back to the user.

[0833] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0834] System Overview

[0835] This invention relates to an AI butler system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses accordingly. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[0836] System configuration

[0837] This system mainly consists of a terminal and a server.

[0838] Terminal

[0839] The device receives voice input from the user and records, converts, compresses, and encodes it. It includes a microphone, a processor, and network communication functions. The device transmits voice data to the server and receives and plays response voice data from the server.

[0840] server

[0841] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from external services and generates a response message in natural language. It also has an emotion recognition engine that analyzes the user's emotions and adjusts the response based on the emotions.

[0842] System Operation

[0843] The operation of the system of the present invention will now be described in detail.

[0844] Brief Description

[0845] 1. Receiving voice input (terminal)

[0846] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[0847] 2. Sending audio data (terminal)

[0848] The digital data is transmitted over a network to a server.

[0849] 3. Speech recognition (server)

[0850] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine, which generates the text "What's the weather like today?"

[0851] 4. Emotion Recognition (Server)

[0852] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, detecting, for example, whether the user is tired or happy.

[0853] 5. Text Analysis and Natural Language Processing (Server)

[0854] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[0855] 6. Information Acquisition (Server)

[0856] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[0857] 7. Generating a Response Message (Server)

[0858] The server generates a natural-sounding response message based on the weather information it has acquired, e.g., "It's sunny today, and the temperature is 25 degrees." It also adjusts the message to an appropriate tone, taking into account the user's emotions.

[0859] 8. Text-to-speech conversion (server)

[0860] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[0861] 9. Sending audio data (server)

[0862] The server transmits the generated voice data to the terminal via the network.

[0863] 10. Response playback (terminal)

[0864] The device decodes and decompresses the received audio data and plays it through the speaker.

[0865] Specific examples

[0866] The operation of this system will be explained below using specific scenarios.

[0867] Example: Obtaining weather information based on emotions

[0868] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[0869] 1. The device records the audio and converts it into digital data.

[0870] 2. Send the digital data to the server.

[0871] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[0872] 4. The server uses an emotion recognition engine to determine that the user is tired.

[0873] 5. The server understands that you are requesting weather information and retrieves the latest weather information from the weather forecast API.

[0874] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[0875] 7. The response message is converted into voice data and sent to the terminal.

[0876] 8. The device plays the audio data and tells the user, "It's sunny today and the temperature is 25 degrees. Good luck."

[0877] In this way, the system can take into account the user's emotions and provide more personalized responses, creating a more natural and comfortable interface for users, enabling them to efficiently search for complex information and perform tasks.

[0878] The processing flow will be explained below.

[0879] Step 1:

[0880] Receiving voice input (user, device)

[0881] The user speaks to the device, saying, "Tell me the weather today." The device's microphone converts this voice into digital data, capturing the voice data.

[0882] Step 2:

[0883] Audio data buffering (terminal)

[0884] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[0885] Step 3:

[0886] Audio data compression and encoding (terminal)

[0887] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[0888] Step 4:

[0889] Sending audio data (terminal)

[0890] The device sends the compressed and encoded audio data to the server via the network.

[0891] Step 5:

[0892] Receiving an HTTP request (server)

[0893] The server receives the HTTP request sent from the terminal and acquires the audio data.

[0894] Step 6:

[0895] Decoding and decompression (server)

[0896] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[0897] Step 7:

[0898] Speech recognition (server)

[0899] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[0900] Step 8:

[0901] Emotion recognition (server)

[0902] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, for example, detecting whether the user is tired or happy from the tone of their voice and speaking patterns.

[0903] Step 9:

[0904] Text analysis and natural language processing (server)

[0905] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[0906] Step 10:

[0907] Information acquisition (server)

[0908] In response to a user request, the server calls the weather forecast API of an external service to obtain current weather information.

[0909] Step 11:

[0910] Generate a response message (server)

[0911] The server generates a natural-sounding response message based on the acquired weather information. For example, the generated message might be, "It's sunny today, and the temperature is 25 degrees." In addition, it adds encouraging words such as "Keep up the good work!" based on the user's emotions (e.g., tiredness) analyzed by an emotion recognition engine.

[0912] Step 12:

[0913] Text-to-speech conversion (server)

[0914] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[0915] Step 13:

[0916] Audio data encoding and compression (server)

[0917] The server compresses and encodes the generated audio data into an efficient transmission format.

[0918] Step 14:

[0919] Sending a response (server)

[0920] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[0921] Step 15:

[0922] Decoding and decompression (terminal)

[0923] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[0924] Step 16:

[0925] Audio playback (device)

[0926] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "It's sunny today, and the temperature is 25 degrees. Good luck!"

[0927] Example 2

[0928] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0929] Conventional speech recognition systems simply convert a user's speech into text and understand their intent, but it is difficult to generate responses that take the user's emotions into account. As a result, many responses are mechanical and ignore the user's emotional state, resulting in a limited user experience. Furthermore, the entire process from voice input to response generation requires immediacy and accuracy of information, but there has been a lack of technology to achieve this.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0931] In this invention, the server includes means including a voice recognition engine for converting voice data into text data, natural language processing and emotion recognition means for analyzing the text data and understanding the user's intentions and emotions, and means for acquiring information from an external database, thereby enabling the server to provide a more personalized response to the user by converting the user's voice input into digital data and analyzing the user's emotions as well as their intentions to generate an appropriate response message.

[0932] "Voice input" refers to an input format in which a user gives instructions or asks questions using voice.

[0933] "Digital data" refers to data obtained by converting analog audio signals into digital format.

[0934] A "server" is a computer system that communicates with other computers and devices over a network and provides specific functions or services.

[0935] A "speech recognition engine" is a piece of software or hardware that converts voice data into text data.

[0936] "Natural language processing" is a technology that enables computers to understand, analyze, and generate human language.

[0937] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data and text data.

[0938] An "external database" is a collection of information that exists outside the system and is a database that is accessed to obtain required information.

[0939] "Text data" is character string data converted from voice input by a voice recognition engine.

[0940] A "response message" is a reply in natural language format that the system generates in response to a user's question or instruction.

[0941] A "text-to-speech engine" is a piece of software or hardware for converting text data into speech data.

[0942] A "terminal" is a device that allows a user to input voice and has the function of communicating with a server.

[0943] "Decoding" is the process of returning encoded data to its original form.

[0944] "Decompression" is the process of restoring compressed data to its original state.

[0945] System Overview

[0946] This invention relates to an AI system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses based on those emotions. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[0947] System configuration

[0948] This system mainly consists of a terminal and a server.

[0949] Terminal

[0950] The device receives voice input from the user and records, converts, compresses, and encodes it. This includes a microphone, a processor, and network communication capabilities. Specifically, the device's microphone (e.g., a microphone) converts the voice into digital data, and the processor (e.g., a central processing unit) compresses and encodes the data. The device then uses its network communication capabilities (e.g., a wireless communication module) to send the data to a server, receive response voice data from the server, and play it back.

[0951] server

[0952] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. For example, the server converts the received voice data into text data using a speech recognition engine (e.g., generic voice recognition software). The server also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server retrieves necessary information from an external database and generates a natural language response message. The server also has an emotion recognition engine (e.g., generic emotion analysis software) that adjusts the response based on the user's emotion.

[0953] System Operation

[0954] Receiving and processing audio input (terminal)

[0955] When a user speaks a voice command into the device, such as "What's the weather today?", the device's microphone converts the voice into digital data, which is then compressed and encoded by a processor and sent to a server via network communications.

[0956] Speech and emotion recognition (server)

[0957] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine. For example, text data such as "What's the weather like today?" is generated. Then, an emotion recognition engine is used to analyze the user's emotions from the voice data. For example, emotions such as tiredness or happiness can be detected.

[0958] Text analysis and information retrieval (server)

[0959] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. For example, it determines that the user is "requesting weather information." The server then calls an external database (e.g., a general-name weather forecast database) to obtain current weather information.

[0960] Generate and send a response message (server)

[0961] The server generates a natural-sounding response message based on the acquired weather information. For example, it might generate a message like "It's sunny today, and the temperature is 25 degrees." Taking the user's emotions into consideration, the server might add words of encouragement, such as "Good luck." The server then converts the generated text response message into audio data using a text-to-speech engine and sends it to the device.

[0962] Response playback (terminal)

[0963] The device decodes and decompresses the received audio data and plays it back through a speaker (e.g., a general-purpose audio output device). Specifically, it conveys to the user something like, "Today is sunny, and the temperature is 25 degrees. Good luck."

[0964] Specific examples

[0965] The operation of this system will be explained below using specific scenarios.

[0966] Example: Obtaining weather information based on emotions

[0967] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[0968] 1. The device records the sound and converts it into digital data (microphone: the common name for a microphone).

[0969] 2. Send the digital data to the server (network communication: commonly known as a wireless communication module).

[0970] 3. The server converts the voice data into text (voice recognition engine: generic name for voice recognition software) and recognizes the text as "Tell me what the weather is today."

[0971] 4. The server uses an emotion recognition engine (commonly known as emotion analysis software) to determine that the user is tired.

[0972] 5. The server understands that you are requesting weather information (natural language processing engine: a general term for natural language processing software) and retrieves the latest weather information from a weather forecast database (a general term for an external database).

[0973] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[0974] 7. The response message is converted into voice data (text-to-speech engine: a common name for speech synthesis software) and sent to the terminal.

[0975] 8. The device plays audio data (speaker: a common name for an audio output device) and tells the user, "It's sunny today, and the temperature is 25 degrees. Good luck."

[0976] Prompt Sentence Examples

[0977] "Tell me about today's weather."

[0978] "Check tomorrow's schedule."

[0979] "Play some uplifting music"

[0980] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0981] Step 1:

[0982] The device receives voice input from the user. When the user says to the device, "What's the weather today?", the microphone converts the voice into digital data. The input is analog voice and the output is digital data. This data is then processed by a processor within the device.

[0983] Step 2:

[0984] The device compresses and encodes the digital data. Specifically, a processor (e.g., a central processing unit) compresses the digital data into MP3 or AAC format and encodes the data into a format suitable for the network. The input is digital audio data, and the output is compressed and encoded audio data.

[0985] Step 3:

[0986] The device sends the compressed and encoded audio data to the server via a network communication function (e.g., a wireless communication module). The data is encrypted to ensure secure communication. The input is the compressed and encoded audio data, and the output is the transmission of the audio data to the server.

[0987] Step 4:

[0988] The server decodes and decompresses the received audio data, specifically using the server's (e.g., computer system's) decoding and decompression capabilities (e.g., generic software) to return the audio data to its original digital form. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data.

[0989] Step 5:

[0990] The server uses a speech recognition engine (e.g., speech recognition software) to convert the decoded and decompressed audio data into text data. Specifically, the speech recognition engine analyzes the audio data and generates the text "What's the weather like today?" The input is the decoded audio data, and the output is text data.

[0991] Step 6:

[0992] The server uses an emotion recognition engine (e.g., emotion analysis software) to analyze the user's emotions from the voice data. Specifically, the emotion recognition engine analyzes the tone and pitch of the voice to identify the user's emotions, such as tiredness or joy. The input is the voice data, and the output is the user's emotional information as a result of the analysis.

[0993] Step 7:

[0994] The server uses a natural language processing engine (e.g., natural language processing software) to analyze the text data and understand the user's intent. Specifically, the natural language processing engine processes the text data and determines that it is a "request for weather information." The input is the text data and the user's emotional information, and the output is the analysis result of the user's intent.

[0995] Step 8:

[0996] The server, according to the user's request, calls an external database (e.g., a weather forecast database) to obtain current weather information. Specifically, the server sends a request to the database via an API to obtain weather information. The input is the user's intent and the API request from the external database, and the output is weather information.

[0997] Step 9:

[0998] The server generates a natural response message based on the acquired weather information. Specifically, the server uses its response message generation function to generate a message such as "It's sunny today, and the temperature is 25 degrees. Good luck!" The input is weather information and the user's emotional information, and the output is the response message.

[0999] Step 10:

[1000] The server uses a text-to-speech engine (e.g., speech synthesis software) to convert the generated text response message into audio data. Specifically, the text-to-speech engine converts the text data into audio data. The input is the response message, and the output is audio data.

[1001] Step 11:

[1002] The server sends the generated voice data to the device via the network communication function. The data is again encrypted to ensure secure communication. The input is voice data, and the output is the transmission of voice data to the device.

[1003] Step 12:

[1004] The device decodes and decompresses the received voice data and plays it back through the speaker. Specifically, the device uses its decoding function to return the voice data to its original format and plays back from the speaker, "It's sunny today, and the temperature is 25 degrees. Good luck." The input is the voice data received from the server, and the output is a voice response to the user.

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1007] To improve passenger comfort in autonomous vehicles, systems that can accurately understand passenger emotions and intentions and provide appropriate responses based on those emotions are required. However, current voice assistant systems for autonomous vehicles lack emotion recognition capabilities and are unable to provide personalized responses based on passenger emotions, which can cause discomfort to passengers.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intention, means for acquiring information from an external service based on the user's intention, means for generating the acquired information into a response message in natural language, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, means for analyzing the user's emotion from the voice data, and means for adjusting the response message based on the user's emotion. This enables passengers in an autonomous vehicle to receive a personalized response based on their emotions.

[1009] The "means for receiving user voice input" is a device or function that receives voice commands from the user through a microphone.

[1010] The "means for converting the voice input into digital data" refers to a device or function that converts analog input voice data into digital data.

[1011] The "means for transmitting the digital data to the server" refers to a device or function that transmits the digital data to the server via network communication.

[1012] The "means including a voice recognition engine for converting said digital data into text data" refers to a device or function that converts digital voice data into text form using voice recognition technology.

[1013] The "natural language processing means for analyzing the text data and understanding the user's intent" is a device or function that uses natural language processing technology to analyze the text data and understand the user's intent.

[1014] The "means for acquiring information from an external service based on the user's intention" is a device or function for acquiring necessary information from an external information providing service based on the analyzed user's intention.

[1015] The "means for generating a response message in a natural language from the acquired information" refers to a device or function that generates a response message in a natural language based on the acquired information.

[1016] The "means including a text-to-speech engine for converting the response message into voice data" refers to a device or function that converts the generated text message into voice data.

[1017] The "means for transmitting the voice data to the terminal" is a device or function for transmitting the generated voice data to the terminal.

[1018] The "means for reproducing the audio data" refers to a device or function such as a speaker that reproduces audio data on the terminal.

[1019] The "means for analyzing the user's emotions from the voice data" refers to a device or function that analyzes the voice data and identifies the user's emotional state.

[1020] The "means for adjusting the response message based on the user's emotions" refers to a device or function that adjusts the content and tone of the response message based on the analyzed emotional state of the user.

[1021] A "means for receiving voice input from a passenger of an automated vehicle" is a device or function within the automated vehicle that receives voice input from a passenger.

[1022] A "generative AI model" refers to a trained artificial intelligence model used for natural language processing and intent analysis.

[1023] A "prompt sentence" is an input sentence containing instructions or questions from a user given to a generative AI model.

[1024]

[1025] This invention relates to a voice assistant system for improving passenger comfort in autonomous vehicles. The system can provide personalized services to passengers by using AI technology to receive user voice input, analyze it, and generate appropriate responses.

[1026] System Configuration

[1027] This system mainly consists of a terminal and a server.

[1028] Terminal

[1029] The terminal is installed inside the autonomous vehicle and has the following functions:

[1030] 1. Audio input reception:

[1031] Passenger voice commands are received via a microphone.

[1032] 2. Audio digital conversion:

[1033] Converts audio input into digital data.

[1034] 3. Data transmission:

[1035] The converted digital data is sent to a server.

[1036] 4. Audio data playback:

[1037] The response voice data sent from the server is played back.

[1038] server

[1039] The server receives the digital audio data sent from the terminal and provides the following functions:

[1040] 1. Speech Recognition Engine:

[1041] Converts digital voice data into text data using the Google Cloud Speech-to-Text API or similar.

[1042] 2. Natural Language Processing Engine:

[1043] The converted text data is analyzed to understand the user's intent. OpenAI's generative AI models (such as GPT-3 / 4) are used.

[1044] 3. Emotion Recognition Engine:

[1045] Analyzes user emotions from voice data, using Azure Emotion API etc.

[1046] 4. Information acquisition means:

[1047] Obtain necessary information from external services based on the user's intent.

[1048] 5. Response message generation:

[1049] A natural language response message is generated based on the acquired information and the results of sentiment analysis.

[1050] 6. Text-to-speech engine:

[1051] The generated text message is converted into audio data, using the Google Text-to-Speech API or similar.

[1052] 7. Means of data transmission:

[1053] The converted audio data is sent to the terminal.

[1054] System Operation

[1055] The system works as follows:

[1056] 1. Receiving voice input:

[1057] The user (passenger) issues a voice command to the terminal, for example, a prompt such as "What's the weather like today?"

[1058] 2. Data transmission:

[1059] The terminal transmits the voice data to the server.

[1060] 3. Speech Recognition and Natural Language Processing:

[1061] The server converts the received voice data into text and analyzes it using a natural language processing engine to understand the user's intent.

[1062] 4. Emotion recognition:

[1063] An emotion recognition engine is used to analyze the user's emotions and tailor the response message based on the emotions.

[1064] 5. Information Acquisition and Response Generation:

[1065] External information is acquired based on the user's intention, and a personalized response message is generated.

[1066] 6. Audio data conversion and transmission:

[1067] The text-to-speech engine converts the response message into voice data, which is then sent to the terminal.

[1068] 7. Response playback:

[1069] The terminal plays back the transmitted voice data and conveys the response to the passenger.

[1070] Specific examples

[1071] For example, if a passenger asks the terminal, "What's the weather like today?", and the passenger sounds a little tired, the system works as follows:

[1072] 1. The device receives the audio, converts it into digital data, and sends it to the server.

[1073] 2. The server uses a speech recognition engine to convert the text into "What's the weather like today?"

[1074] 3. Use an emotion recognition engine to detect tired emotions from voice data.

[1075] 4. The natural language processing engine analyzes the user's intent and understands that they are looking for weather information.

[1076] 5. The server retrieves the latest weather information from the weather forecast API.

[1077] 6. Based on the emotion recognition results, generate a response message such as "It's a sunny day today. Please relax."

[1078] 7. The response message is converted into voice data and sent to the terminal.

[1079] 8. The device plays audio data telling the passenger, "It's a sunny day today. Please relax."

[1080] An example prompt is:

[1081] "Please tell me the weather today."

[1082] "Please tell me the current traffic information."

[1083] This enables the system to provide passengers with personalized services and a more comfortable travel experience.

[1084] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1085] Step 1: Receiving audio input

[1086] The terminal receives voice commands from the user (passenger) using a microphone. The input is the passenger's voice, and the output is analog voice data. Specifically, the microphone captures the voice and converts it into a digital signal.

[1087] Step 2: Sending audio data

[1088] The terminal transmits the converted digital audio data to the server. The input is the digital audio data, and the output is the data transmitted to the server. Specifically, the terminal transmits the data to the server via network communication.

[1089] Step 3: Voice Recognition

[1090] The server converts the received digital voice data into text data using a speech recognition engine. The input is digital voice data, and the output is text data. Specifically, the server calls the Google Cloud Speech-to-Text API to obtain the text data.

[1091] Step 4: Natural Language Processing

[1092] The server analyzes the text data using a natural language processing engine to understand the user's intent. The input is text data, and the output is the analyzed intent. Specifically, the server uses OpenAI's generative AI model (GPT-3 / 4) to analyze the text data and determine the intent.

[1093] Step 5: Emotion Recognition

[1094] The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The input is the voice data, and the output is the user's emotional state. Specifically, the server calls the Azure Emotion API to analyze the voice data and identify the emotion.

[1095] Step 6: Information Acquisition

[1096] The server retrieves the necessary information from external services based on the analyzed intent. The input is the user's intent, and the output is the retrieved information. Specifically, the server retrieves information by calling external weather forecast APIs and traffic information APIs.

[1097] Step 7: Generate a response message

[1098] The server generates a response message in natural language based on the acquired information and the results of emotion analysis. The input is the acquired information and the user's emotional state, and the output is the response message. Specifically, the server uses the generative AI model to create a text message.

[1099] Step 8: Text-to-speech

[1100] The server converts the generated text message into audio data. The input is the response message, and the output is audio data. Specifically, the server calls the Google Text-to-Speech API to convert the text into audio data.

[1101] Step 9: Sending audio data

[1102] The server transmits the converted voice data to the terminal. The input is the voice data, and the output is the data transmitted to the terminal. Specifically, the server transmits the voice data to the terminal via network communication.

[1103] Step 10: Playing back the response

[1104] The terminal plays the received voice data. The input is voice data, and the output is the voice heard by the passenger. Specifically, the voice data is played through the terminal's speaker to convey the response to the passenger.

[1105] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1107] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1108] [Third embodiment]

[1109] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1110] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1111] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1112] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1113] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1115] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1116] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1117] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1118] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1119] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1120] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1121] System Overview

[1122] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[1123] System configuration

[1124] This system mainly consists of a terminal and a server.

[1125] Terminal

[1126] The terminal receives voice input from the user and records, converts, compresses, and encodes the voice. It includes a microphone, a processor, and network communication functions. The terminal transmits the voice data to the server and receives and plays back response voice data from the server.

[1127] server

[1128] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from an external service and generates a response message in natural language. The response message is converted into voice data using a text-to-speech engine and sent to the terminal.

[1129] System Operation

[1130] The operation of the system of the present invention will now be described in detail.

[1131] Brief Description

[1132] 1. Receiving voice input (terminal)

[1133] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[1134] 2. Sending audio data (terminal)

[1135] The digital data is transmitted over a network to a server.

[1136] 3. Speech recognition (server)

[1137] The server decodes and decompresses the received voice data and converts it into text data using a voice recognition engine.

[1138] 4. Natural Language Processing (Server)

[1139] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, obtaining weather information).

[1140] 5. Information Acquisition (Server)

[1141] The server calls external services such as weather forecast APIs to obtain the latest weather information.

[1142] 6. Response Generation (Server)

[1143] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated.

[1144] 7. Voice data generation (server)

[1145] The server converts the response message into audio data using a text-to-speech engine, and encodes and compresses it.

[1146] 8. Sending a Response (Server)

[1147] The generated voice data is transmitted to the terminal via the network.

[1148] 9. Response playback (terminal)

[1149] The device decodes and decompresses the received audio data and plays it through the speaker.

[1150] Specific examples

[1151] The operation of this system will be explained below using specific scenarios.

[1152] Example: Getting weather information

[1153] In the morning, a user speaks to a smart speaker in the living room, saying, "Good morning, tell me today's weather."

[1154] 1. The device records the audio and converts it into digital data.

[1155] 2. Send the digital data to the server.

[1156] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[1157] 4. The server parses this text and realizes that you are requesting weather information.

[1158] 5. Call an external weather forecast API to get current weather information.

[1159] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[1160] 7. Convert the response into audio data, encode it, and send it to the device.

[1161] 8. The device plays the audio data and tells the user the weather information.

[1162] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[1163] The processing flow will be explained below.

[1164] Step 1:

[1165] Receiving voice input (user, device)

[1166] The user speaks to the device, saying, "Tell me today's weather." The device receives the user's voice through a microphone and converts the voice data into digital data.

[1167] Step 2:

[1168] Audio data buffering (terminal)

[1169] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[1170] Step 3:

[1171] Audio data compression and encoding (terminal)

[1172] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[1173] Step 4:

[1174] Sending audio data (terminal)

[1175] The device sends the compressed and encoded audio data to the server via the network.

[1176] Step 5:

[1177] Receiving an HTTP request (server)

[1178] The server receives the HTTP request sent from the terminal and acquires the audio data.

[1179] Step 6:

[1180] Decoding and decompression (server)

[1181] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[1182] Step 7:

[1183] Speech recognition (server)

[1184] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[1185] Step 8:

[1186] Text analysis and natural language processing (server)

[1187] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[1188] Step 9:

[1189] Information acquisition (server)

[1190] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[1191] Step 10:

[1192] Generate a response message (server)

[1193] The server generates a natural response message based on the weather information it retrieves, for example, "It's sunny today and the temperature is 25 degrees."

[1194] Step 11:

[1195] Text-to-speech conversion (server)

[1196] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[1197] Step 12:

[1198] Audio data encoding and compression (server)

[1199] The server compresses and encodes the generated audio data into an efficient transmission format.

[1200] Step 13:

[1201] Sending a response (server)

[1202] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[1203] Step 14:

[1204] Decoding and decompression (terminal)

[1205] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[1206] Step 15:

[1207] Audio playback (device)

[1208] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "Today is sunny and the temperature is 25 degrees."

[1209] Example 1

[1210] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1211] Conventional voice assistant systems have difficulty accurately understanding the user's intent due to the low accuracy of speech recognition and natural language processing. They also suffer from poor response generation quality and insufficient information provided to the user. Furthermore, communication delays and the time required for voice data compression and encoding often impair real-time performance.

[1212] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1213] In this invention, the server includes means including a voice recognition engine that converts voice data into text data, natural language processing means that analyzes the text data and understands the user's intention, means for acquiring information from an external service based on the user's intention, means including a generative AI model that generates a response message in natural language from the acquired information, and means for transmitting the voice data to the terminal, thereby enabling a quick and accurate response to the user's voice command to be provided.

[1214] "Voice input" refers to voice commands or questions spoken by the user into a microphone.

[1215] "Converting to digital data" refers to the process of converting analog audio signals into digital format data.

[1216] "Compression and encoding" refers to the process of reducing the size of digital data and converting it into a particular format.

[1217] "Server" means a computer system that receives, processes, and transmits audio data over a network.

[1218] A "voice recognition engine" refers to software or a system for converting voice data into text data.

[1219] "Natural language processing means" refers to software or a system that analyzes text data and understands the user's intent.

[1220] A "generative AI model" refers to a machine learning model for generating natural language response messages.

[1221] "Text-to-speech engine" refers to software or a system for converting text data into speech data.

[1222] "Response message" refers to a natural language message generated in response to a user's voice command.

[1223] "Means of obtaining information from external services" refers to the systems or means for obtaining the necessary information from external data sources or APIs.

[1224] "Means for playing audio data" refers to a system or device for decoding text-to-speech data and providing it to a user as audio.

[1225] "Terminal" refers to the device through which the user inputs voice and receives and plays responses from the server.

[1226] MODE FOR CARRYING OUT THE INVENTION

[1227] System Overview

[1228] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[1229] System configuration

[1230] This system mainly consists of a terminal and a server.

[1231] Terminal

[1232] The device receives voice input from the user and records, converts, compresses, and encodes the voice. This includes a microphone, a processor, and network communication functions. The device transmits the voice data to a server and receives and plays back response voice data from the server. Specific devices include smart speakers and smartphones.

[1233] server

[1234] The server includes a speech recognition engine that receives voice data sent from the device and converts it into text data. It also has natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains the necessary information from external services and generates a natural language response message. The response message is converted into voice data using a text-to-speech engine and sent to the device. Specific technologies used include Google Cloud Speech-to-Text API, Hugging Face Transformers, OpenWeatherMap API, and Amazon Polly.

[1235] System Operation

[1236] 1. Receiving voice input (terminal)

[1237] The user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[1238] 2. Sending audio data (terminal)

[1239] The digital data is transmitted to a server over a network, typically a Wi-Fi network.

[1240] 3. Speech recognition (server)

[1241] The server decodes and decompresses the received audio data and converts it into text using a speech recognition engine, using the Google Cloud Speech-to-Text API.

[1242] 4. Natural Language Processing (Server)

[1243] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, getting weather information). The analysis is performed using Hugging Face Transformers.

[1244] 5. Information Acquisition (Server)

[1245] The server calls external services such as weather forecast APIs to obtain the latest weather information. In this case, we use the OpenWeatherMap API.

[1246] 6. Response Generation (Server)

[1247] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated. GPT-3 can be used as a generative AI model.

[1248] 7. Voice data generation (server)

[1249] The server converts the response message into audio data using a text-to-speech engine, encodes and compresses it, and generates the audio data using Amazon Polly.

[1250] 8. Sending a Response (Server)

[1251] The generated audio data is sent to the device via the network, usually using the HTTP protocol.

[1252] 9. Response playback (terminal)

[1253] The device decodes and decompresses the received audio data and plays it through the speaker.

[1254] Specific examples

[1255] The operation of this system will be explained below using specific scenarios.

[1256] Example: Getting weather information

[1257] In the morning, a user sits in the living room and asks their smart speaker, "Good morning, what's the weather today?"

[1258] 1. The device records the audio and converts it into digital data.

[1259] 2. Send the digital data to the server.

[1260] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[1261] 4. The server parses this text and realizes that you are requesting weather information.

[1262] 5. Call an external weather forecast API to get current weather information.

[1263] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[1264] 7. Convert the response into audio data, encode it, and send it to the device.

[1265] 8. The device plays the audio data and tells the user the weather information.

[1266] Prompt example

[1267] Below are examples of prompt sentences to input to the generative AI model.

[1268] example:

[1269] Provides information related to the user's voice command "What's the weather today?"

[1270] Weather information retrieval prompt:

[1271] User's voice command: "What's the weather like today?"

[1272] Get the response from the weather forecast API and generate a response message in the following format.

[1273] Format: Today is {weather} and the temperature is {temperature} degrees.

[1274] For example: "It's sunny today and the temperature is 25 degrees."

[1275] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[1276] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1277] Step 1: Receiving Audio Input (Device)

[1278] A user issues a voice command to a device (such as a smart speaker). For example, if the user says, "Tell me the weather today," the device's microphone receives this voice. The input is the user's voice, and the output is an analog voice signal. This analog signal is converted into digital data by a processor.

[1279] Step 2: Convert and encode audio data (device)

[1280] The terminal converts the received analog voice signal into digital data. This digital data is then compressed and encoded. The input is an analog voice signal, and the output is compressed and encoded digital data. This process uses an audio codec (e.g., AAC or MP3).

[1281] Step 3: Sending audio data (terminal)

[1282] The device sends the compressed and encoded audio data to the server using a network communication function (such as Wi-Fi). The input is the compressed and encoded digital data, and the output is an HTTP request sent to the server.

[1283] Step 4: Receive and decode the audio data (server)

[1284] The server receives the audio data sent from the device. The received data is decoded and decompressed. The input is compressed and encoded digital data, and the output is decoded and decompressed digital audio data. A decoder (e.g., AAC decoder) is used for this process.

[1285] Step 5: Speech Recognition (Server)

[1286] The server converts the decoded and decompressed audio data into text data using a speech recognition engine (e.g., speech recognition API). The input is the decoded and decompressed digital audio data, and the output is text data such as "What's the weather like today?"

[1287] Step 6: Natural Language Processing (Server)

[1288] The server analyzes the text data obtained by speech recognition using a natural language processing engine (e.g., a natural language processing library) to understand the user's intent. The input is text data, and the output is the analyzed user's intent (e.g., getting weather information).

[1289] Step 7: Information Acquisition (Server)

[1290] The server retrieves the necessary information (e.g., weather information) from an external service based on the analyzed user intent. For example, it calls a weather forecast API to retrieve current weather information. The input is the user intent, and the output is the weather information retrieved from the external service (e.g., sunny, temperature 25 degrees).

[1291] Step 8: Response Generation (Server)

[1292] The server uses a generative AI model to generate a natural language response message based on the acquired information. For example, it generates a message such as "It's sunny today, and the temperature is 25 degrees." The input is weather information, and the output is the text data of the response message.

[1293] Step 9: Generate audio data (server)

[1294] The server converts the generated response message into audio data using a text-to-speech engine, and encodes and compresses it. The input is the text data of the response message, and the output is the compressed and encoded audio data.

[1295] Step 10: Sending the Response (Server)

[1296] The server sends the generated audio data to the device via the network. The input is the compressed and encoded audio data, and the output is the transmission of an HTTP response to the device.

[1297] Step 11: Receive and Play Response (Terminal)

[1298] The device receives the audio data sent from the server, decodes and decompresses it. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data. It then plays the audio data through the device's speaker and provides the user with a voice response saying, "Today is sunny and the temperature is 25 degrees."

[1299] (Application example 1)

[1300] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1301] Traditionally, customer service in physical stores has relied on a limited number of staff, making it difficult to provide fast and accurate information, especially during busy times. Furthermore, when customers wanted information about a product, they had to search for it themselves, which took time and effort. This could lead to a decline in customer satisfaction. Therefore, there is a demand for a system that allows customers to easily obtain product information through voice input.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1303] In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intent, means for acquiring information from an external service based on the user's intent, means for generating a response message in natural language from the acquired information, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, and the system is used to provide product information in a physical store, means for acquiring information from a product database based on the user's intent, and means for generating and playing the acquired information as a response message. This enables customers to quickly and accurately obtain product information through voice input.

[1304] "Voice input" is a means by which a user provides instructions or information to a system using voice.

[1305] "Digital data" refers to data that has been converted from analog audio information into a digital format.

[1306] A "server" is a computer system that provides various services over a network.

[1307] A "voice recognition engine" is software that analyzes input voice data and converts it into text data.

[1308] "Natural language processing means" refers to techniques and methods for analyzing text data and understanding user intent.

[1309] "External services" are other systems or APIs that the server uses in response to user requests.

[1310] A "response message" is a text response generated based on the user's intent.

[1311] A "text-to-speech engine" is software for converting text data into audio data.

[1312] A "terminal" is a hardware device that is directly operated by a user and receives and plays audio.

[1313] A "product database" is a collection of data that collects and stores product information from physical stores.

[1314] A "physical store" is a store that sells products at a physical location.

[1315] "Customers" refer to typical consumers who use physical stores.

[1316] System Overview

[1317] This invention relates to a system that enables customers in a physical store to quickly and accurately obtain product information through voice input. The system is primarily composed of hardware and software for receiving voice input, converting it to digital data, transmitting it to a server, performing voice recognition, natural language processing, information acquisition, response generation, and voice output.

[1318] Hardware and software used

[1319] Terminal: Smartphone or smart device (including microphone, processor, and network communication function)

[1320] Server: Speech recognition engine, natural language processing engine, text-to-speech engine

[1321] External service: Product database access API

[1322] Operating procedure

[1323] Usage example

[1324] Consider a scenario in which a customer uses a smartphone in a physical store to ask about product information. The customer speaks into the smartphone's microphone, saying, "Tell me the price of this product."

[1325] 1. Receiving voice input (terminal): The user speaks into the smartphone's microphone. The terminal converts this voice into digital data, compresses it, and encodes it.

[1326] 2. Sending digital data (terminal): The terminal sends the digital data to the server via network communication.

[1327] 3. Speech Recognition (Server): The server decodes and decompresses the received digital data and converts it into text data using a speech recognition engine.

[1328] 4. Natural Language Processing (Server): The server analyzes the text data and recognizes the customer's request (in this case, obtaining the product price).

[1329] 5. Information acquisition (server): The server calls the API to access the product database and acquires the price information of the relevant product.

[1330] 6. Response generation (server): Based on the obtained price information, a response message such as "The price of this product is XX yen" is generated.

[1331] 7. Generate voice data (Server): The server converts the response message into voice data using a text-to-speech engine, and encodes and compresses it.

[1332] 8. Sending response (server): The server sends the generated voice data to the terminal via the network.

[1333] 9. Playback of response (terminal): The terminal decodes and decompresses the received voice data and plays it back through the speaker. The customer receives a voice response such as "The price of this product is XX yen."

[1334] Specific examples

[1335] Examples of voice commands include prompts such as:

[1336] "Please tell me the price of this item."

[1337] "Tell me what products you recommend."

[1338] "What are the features of this product?"

[1339] When a customer says to their smartphone, "Tell me the price of this item," the system receives the voice input and quickly provides the product price information via voice, allowing customers to get the information they need without any hassle while in the store.

[1340] This is a specific embodiment for carrying out the invention.

[1341] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1342] Step 1:

[1343] The user provides voice input

[1344] The user speaks into the smartphone's microphone, saying, "Tell me the price of this item." The smartphone receives the voice data. In this process, the user's voice is input as an analog signal.

[1345] Step 2:

[1346] Converting audio into digital data

[1347] The terminal converts the received voice into digital data, which includes the processes of voice sampling and quantization, and outputs the analog voice data as digital data.

[1348] Step 3:

[1349] Sending digital data to a server

[1350] The terminal compresses and encodes the converted digital data and sends it to the server via network communication. The input is the converted digital data and the output is the compressed data sent to the server.

[1351] Step 4:

[1352] Converting audio data into text data

[1353] The server decodes and decompresses the received data and converts the voice data into text data using a speech recognition engine. The input is compressed voice data and the output is text data.

[1354] Step 5:

[1355] Analyzing text data to understand user intent

[1356] The server analyzes the text data and uses a natural language processing engine to understand the user's intent. In this process, the input is the text data to be analyzed, and the output is the intent data, such as "I want to know the price of this product."

[1357] Step 6:

[1358] Get product information

[1359] Based on the user's intent, the server calls the API to access the product database and obtains the price information of the relevant product. The input is the understood user intent, and the output is the obtained price information.

[1360] Step 7:

[1361] Generate a response message

[1362] The server generates a response message in natural language based on the acquired product information. The input for this process is product price information, and the output is a response message saying, "The price of this product is XX yen."

[1363] Step 8:

[1364] Converting response messages into audio data

[1365] The server converts the generated response message into voice data using a text-to-speech engine. The input is the response message, and the output is the voice data.

[1366] Step 9:

[1367] Sending audio data to the device

[1368] The server compresses and encodes the converted audio data and sends it to the terminal via network communication. The input is the generated audio data, and the output is the compressed data sent to the terminal.

[1369] Step 10:

[1370] Playing audio data

[1371] The terminal decodes and decompresses the received audio data and plays it through a speaker. The input is compressed audio data and the output is audio information played back to the user.

[1372] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1373] System Overview

[1374] This invention relates to an AI butler system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses accordingly. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[1375] System configuration

[1376] This system mainly consists of a terminal and a server.

[1377] Terminal

[1378] The device receives voice input from the user and records, converts, compresses, and encodes it. It includes a microphone, a processor, and network communication functions. The device transmits voice data to the server and receives and plays response voice data from the server.

[1379] server

[1380] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from external services and generates a response message in natural language. It also has an emotion recognition engine that analyzes the user's emotions and adjusts the response based on the emotions.

[1381] System Operation

[1382] The operation of the system of the present invention will now be described in detail.

[1383] Brief Description

[1384] 1. Receiving voice input (terminal)

[1385] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[1386] 2. Sending audio data (terminal)

[1387] The digital data is transmitted over a network to a server.

[1388] 3. Speech recognition (server)

[1389] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine, which generates the text "What's the weather like today?"

[1390] 4. Emotion Recognition (Server)

[1391] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, detecting, for example, whether the user is tired or happy.

[1392] 5. Text Analysis and Natural Language Processing (Server)

[1393] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[1394] 6. Information Acquisition (Server)

[1395] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[1396] 7. Generating a Response Message (Server)

[1397] The server generates a natural-sounding response message based on the weather information it has acquired, e.g., "It's sunny today, and the temperature is 25 degrees." It also adjusts the message to an appropriate tone, taking into account the user's emotions.

[1398] 8. Text-to-speech conversion (server)

[1399] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[1400] 9. Sending audio data (server)

[1401] The server transmits the generated voice data to the terminal via the network.

[1402] 10. Response playback (terminal)

[1403] The device decodes and decompresses the received audio data and plays it through the speaker.

[1404] Specific examples

[1405] The operation of this system will be explained below using specific scenarios.

[1406] Example: Obtaining weather information based on emotions

[1407] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[1408] 1. The device records the audio and converts it into digital data.

[1409] 2. Send the digital data to the server.

[1410] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[1411] 4. The server uses an emotion recognition engine to determine that the user is tired.

[1412] 5. The server understands that you are requesting weather information and retrieves the latest weather information from the weather forecast API.

[1413] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[1414] 7. The response message is converted into voice data and sent to the terminal.

[1415] 8. The device plays the audio data and tells the user, "It's sunny today and the temperature is 25 degrees. Good luck."

[1416] In this way, the system can take into account the user's emotions and provide more personalized responses, creating a more natural and comfortable interface for users, enabling them to efficiently search for complex information and perform tasks.

[1417] The processing flow will be explained below.

[1418] Step 1:

[1419] Receiving voice input (user, device)

[1420] The user speaks to the device, saying, "Tell me the weather today." The device's microphone converts this voice into digital data, capturing the voice data.

[1421] Step 2:

[1422] Audio data buffering (terminal)

[1423] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[1424] Step 3:

[1425] Audio data compression and encoding (terminal)

[1426] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[1427] Step 4:

[1428] Sending audio data (terminal)

[1429] The device sends the compressed and encoded audio data to the server via the network.

[1430] Step 5:

[1431] Receiving an HTTP request (server)

[1432] The server receives the HTTP request sent from the terminal and acquires the audio data.

[1433] Step 6:

[1434] Decoding and decompression (server)

[1435] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[1436] Step 7:

[1437] Speech recognition (server)

[1438] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[1439] Step 8:

[1440] Emotion recognition (server)

[1441] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, for example, detecting whether the user is tired or happy from the tone of their voice and speaking patterns.

[1442] Step 9:

[1443] Text analysis and natural language processing (server)

[1444] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[1445] Step 10:

[1446] Information acquisition (server)

[1447] In response to a user request, the server calls the weather forecast API of an external service to obtain current weather information.

[1448] Step 11:

[1449] Generate a response message (server)

[1450] The server generates a natural-sounding response message based on the acquired weather information. For example, the generated message might be, "It's sunny today, and the temperature is 25 degrees." In addition, it adds encouraging words such as "Keep up the good work!" based on the user's emotions (e.g., tiredness) analyzed by an emotion recognition engine.

[1451] Step 12:

[1452] Text-to-speech conversion (server)

[1453] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[1454] Step 13:

[1455] Audio data encoding and compression (server)

[1456] The server compresses and encodes the generated audio data into an efficient transmission format.

[1457] Step 14:

[1458] Sending a response (server)

[1459] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[1460] Step 15:

[1461] Decoding and decompression (terminal)

[1462] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[1463] Step 16:

[1464] Audio playback (device)

[1465] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "It's sunny today, and the temperature is 25 degrees. Good luck!"

[1466] Example 2

[1467] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1468] Conventional speech recognition systems simply convert a user's speech into text and understand their intent, but it is difficult to generate responses that take the user's emotions into account. As a result, many responses are mechanical and ignore the user's emotional state, resulting in a limited user experience. Furthermore, the entire process from voice input to response generation requires immediacy and accuracy of information, but there has been a lack of technology to achieve this.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1470] In this invention, the server includes means including a voice recognition engine for converting voice data into text data, natural language processing and emotion recognition means for analyzing the text data and understanding the user's intentions and emotions, and means for acquiring information from an external database, thereby enabling the server to provide a more personalized response to the user by converting the user's voice input into digital data and analyzing the user's emotions as well as their intentions to generate an appropriate response message.

[1471] "Voice input" refers to an input format in which a user gives instructions or asks questions using voice.

[1472] "Digital data" refers to data obtained by converting analog audio signals into digital format.

[1473] A "server" is a computer system that communicates with other computers and devices over a network and provides specific functions or services.

[1474] A "speech recognition engine" is a piece of software or hardware that converts voice data into text data.

[1475] "Natural language processing" is a technology that enables computers to understand, analyze, and generate human language.

[1476] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data and text data.

[1477] An "external database" is a collection of information that exists outside the system and is a database that is accessed to obtain required information.

[1478] "Text data" is character string data converted from voice input by a voice recognition engine.

[1479] A "response message" is a reply in natural language format that the system generates in response to a user's question or instruction.

[1480] A "text-to-speech engine" is a piece of software or hardware for converting text data into speech data.

[1481] A "terminal" is a device that allows a user to input voice and has the function of communicating with a server.

[1482] "Decoding" is the process of returning encoded data to its original form.

[1483] "Decompression" is the process of restoring compressed data to its original state.

[1484] System Overview

[1485] This invention relates to an AI system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses based on those emotions. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[1486] System configuration

[1487] This system mainly consists of a terminal and a server.

[1488] Terminal

[1489] The device receives voice input from the user and records, converts, compresses, and encodes it. This includes a microphone, a processor, and network communication capabilities. Specifically, the device's microphone (e.g., a microphone) converts the voice into digital data, and the processor (e.g., a central processing unit) compresses and encodes the data. The device then uses its network communication capabilities (e.g., a wireless communication module) to send the data to a server, receive response voice data from the server, and play it back.

[1490] server

[1491] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. For example, the server converts the received voice data into text data using a speech recognition engine (e.g., generic voice recognition software). The server also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server retrieves necessary information from an external database and generates a natural language response message. The server also has an emotion recognition engine (e.g., generic emotion analysis software) that adjusts the response based on the user's emotion.

[1492] System Operation

[1493] Receiving and processing audio input (terminal)

[1494] When a user speaks a voice command into the device, such as "What's the weather today?", the device's microphone converts the voice into digital data, which is then compressed and encoded by a processor and sent to a server via network communications.

[1495] Speech and emotion recognition (server)

[1496] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine. For example, text data such as "What's the weather like today?" is generated. Then, an emotion recognition engine is used to analyze the user's emotions from the voice data. For example, emotions such as tiredness or happiness can be detected.

[1497] Text analysis and information retrieval (server)

[1498] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. For example, it determines that the user is "requesting weather information." The server then calls an external database (e.g., a general-name weather forecast database) to obtain current weather information.

[1499] Generate and send a response message (server)

[1500] The server generates a natural-sounding response message based on the acquired weather information. For example, it might generate a message like "It's sunny today, and the temperature is 25 degrees." Taking the user's emotions into consideration, the server might add words of encouragement, such as "Good luck." The server then converts the generated text response message into audio data using a text-to-speech engine and sends it to the device.

[1501] Response playback (terminal)

[1502] The device decodes and decompresses the received audio data and plays it back through a speaker (e.g., a general-purpose audio output device). Specifically, it conveys to the user something like, "Today is sunny, and the temperature is 25 degrees. Good luck."

[1503] Specific examples

[1504] The operation of this system will be explained below using specific scenarios.

[1505] Example: Obtaining weather information based on emotions

[1506] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[1507] 1. The device records the sound and converts it into digital data (microphone: the common name for a microphone).

[1508] 2. Send the digital data to the server (network communication: commonly known as a wireless communication module).

[1509] 3. The server converts the voice data into text (voice recognition engine: generic name for voice recognition software) and recognizes the text as "Tell me what the weather is today."

[1510] 4. The server uses an emotion recognition engine (commonly known as emotion analysis software) to determine that the user is tired.

[1511] 5. The server understands that you are requesting weather information (natural language processing engine: a general term for natural language processing software) and retrieves the latest weather information from a weather forecast database (a general term for an external database).

[1512] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[1513] 7. The response message is converted into voice data (text-to-speech engine: a common name for speech synthesis software) and sent to the terminal.

[1514] 8. The device plays audio data (speaker: a common name for an audio output device) and tells the user, "It's sunny today, and the temperature is 25 degrees. Good luck."

[1515] Prompt Sentence Examples

[1516] "Tell me about today's weather."

[1517] "Check tomorrow's schedule."

[1518] "Play some uplifting music"

[1519] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1520] Step 1:

[1521] The device receives voice input from the user. When the user says to the device, "What's the weather today?", the microphone converts the voice into digital data. The input is analog voice and the output is digital data. This data is then processed by a processor within the device.

[1522] Step 2:

[1523] The device compresses and encodes the digital data. Specifically, a processor (e.g., a central processing unit) compresses the digital data into MP3 or AAC format and encodes the data into a format suitable for the network. The input is digital audio data, and the output is compressed and encoded audio data.

[1524] Step 3:

[1525] The device sends the compressed and encoded audio data to the server via a network communication function (e.g., a wireless communication module). The data is encrypted to ensure secure communication. The input is the compressed and encoded audio data, and the output is the transmission of the audio data to the server.

[1526] Step 4:

[1527] The server decodes and decompresses the received audio data, specifically using the server's (e.g., computer system's) decoding and decompression capabilities (e.g., generic software) to return the audio data to its original digital form. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data.

[1528] Step 5:

[1529] The server uses a speech recognition engine (e.g., speech recognition software) to convert the decoded and decompressed audio data into text data. Specifically, the speech recognition engine analyzes the audio data and generates the text "What's the weather like today?" The input is the decoded audio data, and the output is text data.

[1530] Step 6:

[1531] The server uses an emotion recognition engine (e.g., emotion analysis software) to analyze the user's emotions from the voice data. Specifically, the emotion recognition engine analyzes the tone and pitch of the voice to identify the user's emotions, such as tiredness or joy. The input is the voice data, and the output is the user's emotional information as a result of the analysis.

[1532] Step 7:

[1533] The server uses a natural language processing engine (e.g., natural language processing software) to analyze the text data and understand the user's intent. Specifically, the natural language processing engine processes the text data and determines that it is a "request for weather information." The input is the text data and the user's emotional information, and the output is the analysis result of the user's intent.

[1534] Step 8:

[1535] The server, according to the user's request, calls an external database (e.g., a weather forecast database) to obtain current weather information. Specifically, the server sends a request to the database via an API to obtain weather information. The input is the user's intent and the API request from the external database, and the output is weather information.

[1536] Step 9:

[1537] The server generates a natural response message based on the acquired weather information. Specifically, the server uses its response message generation function to generate a message such as "It's sunny today, and the temperature is 25 degrees. Good luck!" The input is weather information and the user's emotional information, and the output is the response message.

[1538] Step 10:

[1539] The server uses a text-to-speech engine (e.g., speech synthesis software) to convert the generated text response message into audio data. Specifically, the text-to-speech engine converts the text data into audio data. The input is the response message, and the output is audio data.

[1540] Step 11:

[1541] The server sends the generated voice data to the device via the network communication function. The data is again encrypted to ensure secure communication. The input is voice data, and the output is the transmission of voice data to the device.

[1542] Step 12:

[1543] The device decodes and decompresses the received voice data and plays it back through the speaker. Specifically, the device uses its decoding function to return the voice data to its original format and plays back from the speaker, "It's sunny today, and the temperature is 25 degrees. Good luck." The input is the voice data received from the server, and the output is a voice response to the user.

[1544] (Application example 2)

[1545] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1546] To improve passenger comfort in autonomous vehicles, systems that can accurately understand passenger emotions and intentions and provide appropriate responses based on those emotions are required. However, current voice assistant systems for autonomous vehicles lack emotion recognition capabilities and are unable to provide personalized responses based on passenger emotions, which can cause discomfort to passengers.

[1547] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intention, means for acquiring information from an external service based on the user's intention, means for generating the acquired information into a response message in natural language, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, means for analyzing the user's emotion from the voice data, and means for adjusting the response message based on the user's emotion. This enables passengers in an autonomous vehicle to receive a personalized response based on their emotions.

[1548] The "means for receiving user voice input" is a device or function that receives voice commands from the user through a microphone.

[1549] The "means for converting the voice input into digital data" refers to a device or function that converts analog input voice data into digital data.

[1550] The "means for transmitting the digital data to the server" refers to a device or function that transmits the digital data to the server via network communication.

[1551] The "means including a voice recognition engine for converting said digital data into text data" refers to a device or function that converts digital voice data into text form using voice recognition technology.

[1552] The "natural language processing means for analyzing the text data and understanding the user's intent" is a device or function that uses natural language processing technology to analyze the text data and understand the user's intent.

[1553] The "means for acquiring information from an external service based on the user's intention" is a device or function for acquiring necessary information from an external information providing service based on the analyzed user's intention.

[1554] The "means for generating a response message in a natural language from the acquired information" refers to a device or function that generates a response message in a natural language based on the acquired information.

[1555] The "means including a text-to-speech engine for converting the response message into voice data" refers to a device or function that converts the generated text message into voice data.

[1556] The "means for transmitting the voice data to the terminal" is a device or function for transmitting the generated voice data to the terminal.

[1557] The "means for reproducing the audio data" refers to a device or function such as a speaker that reproduces audio data on the terminal.

[1558] The "means for analyzing the user's emotions from the voice data" refers to a device or function that analyzes the voice data and identifies the user's emotional state.

[1559] The "means for adjusting the response message based on the user's emotions" refers to a device or function that adjusts the content and tone of the response message based on the analyzed emotional state of the user.

[1560] A "means for receiving voice input from a passenger of an automated vehicle" is a device or function within the automated vehicle that receives voice input from a passenger.

[1561] A "generative AI model" refers to a trained artificial intelligence model used for natural language processing and intent analysis.

[1562] A "prompt sentence" is an input sentence containing instructions or questions from a user given to a generative AI model.

[1563]

[1564] This invention relates to a voice assistant system for improving passenger comfort in autonomous vehicles. The system can provide personalized services to passengers by using AI technology to receive user voice input, analyze it, and generate appropriate responses.

[1565] System Configuration

[1566] This system mainly consists of a terminal and a server.

[1567] Terminal

[1568] The terminal is installed inside the autonomous vehicle and has the following functions:

[1569] 1. Audio input reception:

[1570] Passenger voice commands are received via a microphone.

[1571] 2. Audio digital conversion:

[1572] Converts audio input into digital data.

[1573] 3. Data transmission:

[1574] The converted digital data is sent to a server.

[1575] 4. Audio data playback:

[1576] The response voice data sent from the server is played back.

[1577] server

[1578] The server receives the digital audio data sent from the terminal and provides the following functions:

[1579] 1. Speech Recognition Engine:

[1580] Converts digital voice data into text data using the Google Cloud Speech-to-Text API or similar.

[1581] 2. Natural Language Processing Engine:

[1582] The converted text data is analyzed to understand the user's intent. OpenAI's generative AI models (such as GPT-3 / 4) are used.

[1583] 3. Emotion Recognition Engine:

[1584] Analyzes user emotions from voice data, using Azure Emotion API etc.

[1585] 4. Information acquisition means:

[1586] Obtain necessary information from external services based on the user's intent.

[1587] 5. Response message generation:

[1588] A natural language response message is generated based on the acquired information and the results of sentiment analysis.

[1589] 6. Text-to-speech engine:

[1590] The generated text message is converted into audio data, using the Google Text-to-Speech API or similar.

[1591] 7. Means of data transmission:

[1592] The converted audio data is sent to the terminal.

[1593] System Operation

[1594] The system works as follows:

[1595] 1. Receiving voice input:

[1596] The user (passenger) issues a voice command to the terminal, for example, a prompt such as "What's the weather like today?"

[1597] 2. Data transmission:

[1598] The terminal transmits the voice data to the server.

[1599] 3. Speech Recognition and Natural Language Processing:

[1600] The server converts the received voice data into text and analyzes it using a natural language processing engine to understand the user's intent.

[1601] 4. Emotion recognition:

[1602] An emotion recognition engine is used to analyze the user's emotions and tailor the response message based on the emotions.

[1603] 5. Information Acquisition and Response Generation:

[1604] External information is acquired based on the user's intention, and a personalized response message is generated.

[1605] 6. Audio data conversion and transmission:

[1606] The text-to-speech engine converts the response message into voice data, which is then sent to the terminal.

[1607] 7. Response playback:

[1608] The terminal plays back the transmitted voice data and conveys the response to the passenger.

[1609] Specific examples

[1610] For example, if a passenger asks the terminal, "What's the weather like today?", and the passenger sounds a little tired, the system works as follows:

[1611] 1. The device receives the audio, converts it into digital data, and sends it to the server.

[1612] 2. The server uses a speech recognition engine to convert the text into "What's the weather like today?"

[1613] 3. Use an emotion recognition engine to detect tired emotions from voice data.

[1614] 4. The natural language processing engine analyzes the user's intent and understands that they are looking for weather information.

[1615] 5. The server retrieves the latest weather information from the weather forecast API.

[1616] 6. Based on the emotion recognition results, generate a response message such as "It's a sunny day today. Please relax."

[1617] 7. The response message is converted into voice data and sent to the terminal.

[1618] 8. The device plays audio data telling the passenger, "It's a sunny day today. Please relax."

[1619] An example prompt is:

[1620] "Please tell me the weather today."

[1621] "Please tell me the current traffic information."

[1622] This enables the system to provide passengers with personalized services and a more comfortable travel experience.

[1623] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1624] Step 1: Receiving audio input

[1625] The terminal receives voice commands from the user (passenger) using a microphone. The input is the passenger's voice, and the output is analog voice data. Specifically, the microphone captures the voice and converts it into a digital signal.

[1626] Step 2: Sending audio data

[1627] The terminal transmits the converted digital audio data to the server. The input is the digital audio data, and the output is the data transmitted to the server. Specifically, the terminal transmits the data to the server via network communication.

[1628] Step 3: Voice Recognition

[1629] The server converts the received digital voice data into text data using a speech recognition engine. The input is digital voice data, and the output is text data. Specifically, the server calls the Google Cloud Speech-to-Text API to obtain the text data.

[1630] Step 4: Natural Language Processing

[1631] The server analyzes the text data using a natural language processing engine to understand the user's intent. The input is text data, and the output is the analyzed intent. Specifically, the server uses OpenAI's generative AI model (GPT-3 / 4) to analyze the text data and determine the intent.

[1632] Step 5: Emotion Recognition

[1633] The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The input is the voice data, and the output is the user's emotional state. Specifically, the server calls the Azure Emotion API to analyze the voice data and identify the emotion.

[1634] Step 6: Information Acquisition

[1635] The server retrieves the necessary information from external services based on the analyzed intent. The input is the user's intent, and the output is the retrieved information. Specifically, the server retrieves information by calling external weather forecast APIs and traffic information APIs.

[1636] Step 7: Generate a response message

[1637] The server generates a response message in natural language based on the acquired information and the results of emotion analysis. The input is the acquired information and the user's emotional state, and the output is the response message. Specifically, the server uses the generative AI model to create a text message.

[1638] Step 8: Text-to-speech

[1639] The server converts the generated text message into audio data. The input is the response message, and the output is audio data. Specifically, the server calls the Google Text-to-Speech API to convert the text into audio data.

[1640] Step 9: Sending audio data

[1641] The server transmits the converted voice data to the terminal. The input is the voice data, and the output is the data transmitted to the terminal. Specifically, the server transmits the voice data to the terminal via network communication.

[1642] Step 10: Playing back the response

[1643] The terminal plays the received voice data. The input is voice data, and the output is the voice heard by the passenger. Specifically, the voice data is played through the terminal's speaker to convey the response to the passenger.

[1644] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1645] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1646] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1647] [Fourth embodiment]

[1648] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1649] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1650] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1651] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1652] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1653] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1654] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1655] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1656] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1657] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1658] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1659] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1660] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1661] System Overview

[1662] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[1663] System configuration

[1664] This system mainly consists of a terminal and a server.

[1665] Terminal

[1666] The terminal receives voice input from the user and records, converts, compresses, and encodes the voice. It includes a microphone, a processor, and network communication functions. The terminal transmits the voice data to the server and receives and plays back response voice data from the server.

[1667] server

[1668] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from an external service and generates a response message in natural language. The response message is converted into voice data using a text-to-speech engine and sent to the terminal.

[1669] System Operation

[1670] The operation of the system of the present invention will now be described in detail.

[1671] Brief Description

[1672] 1. Receiving voice input (terminal)

[1673] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[1674] 2. Sending audio data (terminal)

[1675] The digital data is transmitted over a network to a server.

[1676] 3. Speech recognition (server)

[1677] The server decodes and decompresses the received voice data and converts it into text data using a voice recognition engine.

[1678] 4. Natural Language Processing (Server)

[1679] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, obtaining weather information).

[1680] 5. Information Acquisition (Server)

[1681] The server calls external services such as weather forecast APIs to obtain the latest weather information.

[1682] 6. Response Generation (Server)

[1683] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated.

[1684] 7. Voice data generation (server)

[1685] The server converts the response message into audio data using a text-to-speech engine, and encodes and compresses it.

[1686] 8. Sending a Response (Server)

[1687] The generated voice data is transmitted to the terminal via the network.

[1688] 9. Response playback (terminal)

[1689] The device decodes and decompresses the received audio data and plays it through the speaker.

[1690] Specific examples

[1691] The operation of this system will be explained below using specific scenarios.

[1692] Example: Getting weather information

[1693] In the morning, a user speaks to a smart speaker in the living room, saying, "Good morning, tell me today's weather."

[1694] 1. The device records the audio and converts it into digital data.

[1695] 2. Send the digital data to the server.

[1696] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[1697] 4. The server parses this text and realizes that you are requesting weather information.

[1698] 5. Call an external weather forecast API to get current weather information.

[1699] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[1700] 7. Convert the response into audio data, encode it, and send it to the device.

[1701] 8. The device plays the audio data and tells the user the weather information.

[1702] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[1703] The processing flow will be explained below.

[1704] Step 1:

[1705] Receiving voice input (user, device)

[1706] The user speaks to the device, saying, "Tell me today's weather." The device receives the user's voice through a microphone and converts the voice data into digital data.

[1707] Step 2:

[1708] Audio data buffering (terminal)

[1709] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[1710] Step 3:

[1711] Audio data compression and encoding (terminal)

[1712] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[1713] Step 4:

[1714] Sending audio data (terminal)

[1715] The device sends the compressed and encoded audio data to the server via the network.

[1716] Step 5:

[1717] Receiving an HTTP request (server)

[1718] The server receives the HTTP request sent from the terminal and acquires the audio data.

[1719] Step 6:

[1720] Decoding and decompression (server)

[1721] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[1722] Step 7:

[1723] Speech recognition (server)

[1724] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[1725] Step 8:

[1726] Text analysis and natural language processing (server)

[1727] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[1728] Step 9:

[1729] Information acquisition (server)

[1730] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[1731] Step 10:

[1732] Generate a response message (server)

[1733] The server generates a natural response message based on the weather information it retrieves, for example, "It's sunny today and the temperature is 25 degrees."

[1734] Step 11:

[1735] Text-to-speech conversion (server)

[1736] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[1737] Step 12:

[1738] Audio data encoding and compression (server)

[1739] The server compresses and encodes the generated audio data into an efficient transmission format.

[1740] Step 13:

[1741] Sending a response (server)

[1742] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[1743] Step 14:

[1744] Decoding and decompression (terminal)

[1745] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[1746] Step 15:

[1747] Audio playback (device)

[1748] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "Today is sunny and the temperature is 25 degrees."

[1749] Example 1

[1750] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1751] Conventional voice assistant systems have difficulty accurately understanding the user's intent due to the low accuracy of speech recognition and natural language processing. They also suffer from poor response generation quality and insufficient information provided to the user. Furthermore, communication delays and the time required for voice data compression and encoding often impair real-time performance.

[1752] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1753] In this invention, the server includes means including a voice recognition engine that converts voice data into text data, natural language processing means that analyzes the text data and understands the user's intention, means for acquiring information from an external service based on the user's intention, means including a generative AI model that generates a response message in natural language from the acquired information, and means for transmitting the voice data to the terminal, thereby enabling a quick and accurate response to the user's voice command to be provided.

[1754] "Voice input" refers to voice commands or questions spoken by the user into a microphone.

[1755] "Converting to digital data" refers to the process of converting analog audio signals into digital format data.

[1756] "Compression and encoding" refers to the process of reducing the size of digital data and converting it into a particular format.

[1757] "Server" means a computer system that receives, processes, and transmits audio data over a network.

[1758] A "voice recognition engine" refers to software or a system for converting voice data into text data.

[1759] "Natural language processing means" refers to software or a system that analyzes text data and understands the user's intent.

[1760] A "generative AI model" refers to a machine learning model for generating natural language response messages.

[1761] "Text-to-speech engine" refers to software or a system for converting text data into speech data.

[1762] "Response message" refers to a natural language message generated in response to a user's voice command.

[1763] "Means of obtaining information from external services" refers to the systems or means for obtaining the necessary information from external data sources or APIs.

[1764] "Means for playing audio data" refers to a system or device for decoding text-to-speech data and providing it to a user as audio.

[1765] "Terminal" refers to the device through which the user inputs voice and receives and plays responses from the server.

[1766] MODE FOR CARRYING OUT THE INVENTION

[1767] System Overview

[1768] The present invention relates to an AI butler system that receives a user's voice input, analyzes it, generates an appropriate response, and replies by voice. This system effectively performs speech recognition, natural language processing, information acquisition, response generation, and voice output based on the user's voice command.

[1769] System configuration

[1770] This system mainly consists of a terminal and a server.

[1771] Terminal

[1772] The device receives voice input from the user and records, converts, compresses, and encodes the voice. This includes a microphone, a processor, and network communication functions. The device transmits the voice data to a server and receives and plays back response voice data from the server. Specific devices include smart speakers and smartphones.

[1773] server

[1774] The server includes a speech recognition engine that receives voice data sent from the device and converts it into text data. It also has natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains the necessary information from external services and generates a natural language response message. The response message is converted into voice data using a text-to-speech engine and sent to the device. Specific technologies used include Google Cloud Speech-to-Text API, Hugging Face Transformers, OpenWeatherMap API, and Amazon Polly.

[1775] System Operation

[1776] 1. Receiving voice input (terminal)

[1777] The user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[1778] 2. Sending audio data (terminal)

[1779] The digital data is transmitted to a server over a network, typically a Wi-Fi network.

[1780] 3. Speech recognition (server)

[1781] The server decodes and decompresses the received audio data and converts it into text using a speech recognition engine, using the Google Cloud Speech-to-Text API.

[1782] 4. Natural Language Processing (Server)

[1783] The server extracts the user's intent from the analyzed text data and recognizes the user's request (in this case, getting weather information). The analysis is performed using Hugging Face Transformers.

[1784] 5. Information Acquisition (Server)

[1785] The server calls external services such as weather forecast APIs to obtain the latest weather information. In this case, we use the OpenWeatherMap API.

[1786] 6. Response Generation (Server)

[1787] Based on the weather information obtained, a response message such as "It's sunny today and the temperature is 25 degrees" is generated. GPT-3 can be used as a generative AI model.

[1788] 7. Voice data generation (server)

[1789] The server converts the response message into audio data using a text-to-speech engine, encodes and compresses it, and generates the audio data using Amazon Polly.

[1790] 8. Sending a Response (Server)

[1791] The generated audio data is sent to the device via the network, usually using the HTTP protocol.

[1792] 9. Response playback (terminal)

[1793] The device decodes and decompresses the received audio data and plays it through the speaker.

[1794] Specific examples

[1795] The operation of this system will be explained below using specific scenarios.

[1796] Example: Getting weather information

[1797] In the morning, a user sits in the living room and asks their smart speaker, "Good morning, what's the weather today?"

[1798] 1. The device records the audio and converts it into digital data.

[1799] 2. Send the digital data to the server.

[1800] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[1801] 4. The server parses this text and realizes that you are requesting weather information.

[1802] 5. Call an external weather forecast API to get current weather information.

[1803] 6. Based on the information obtained, generate a response such as "It's sunny today and the temperature is 25 degrees."

[1804] 7. Convert the response into audio data, encode it, and send it to the device.

[1805] 8. The device plays the audio data and tells the user the weather information.

[1806] Prompt example

[1807] Below are examples of prompt sentences to input to the generative AI model.

[1808] example:

[1809] Provides information related to the user's voice command "What's the weather today?"

[1810] Weather information retrieval prompt:

[1811] User's voice command: "What's the weather like today?"

[1812] Get the response from the weather forecast API and generate a response message in the following format.

[1813] Format: Today is {weather} and the temperature is {temperature} degrees.

[1814] For example: "It's sunny today and the temperature is 25 degrees."

[1815] In this way, the system can quickly and accurately provide information based on the user's voice commands, providing an intuitive and easy-to-use interface for users, enabling them to efficiently search for complex information and perform tasks.

[1816] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1817] Step 1: Receiving Audio Input (Device)

[1818] A user issues a voice command to a device (such as a smart speaker). For example, if the user says, "Tell me the weather today," the device's microphone receives this voice. The input is the user's voice, and the output is an analog voice signal. This analog signal is converted into digital data by a processor.

[1819] Step 2: Convert and encode audio data (device)

[1820] The terminal converts the received analog voice signal into digital data. This digital data is then compressed and encoded. The input is an analog voice signal, and the output is compressed and encoded digital data. This process uses an audio codec (e.g., AAC or MP3).

[1821] Step 3: Sending audio data (terminal)

[1822] The device sends the compressed and encoded audio data to the server using a network communication function (such as Wi-Fi). The input is the compressed and encoded digital data, and the output is an HTTP request sent to the server.

[1823] Step 4: Receive and decode the audio data (server)

[1824] The server receives the audio data sent from the device. The received data is decoded and decompressed. The input is compressed and encoded digital data, and the output is decoded and decompressed digital audio data. A decoder (e.g., AAC decoder) is used for this process.

[1825] Step 5: Speech Recognition (Server)

[1826] The server converts the decoded and decompressed audio data into text data using a speech recognition engine (e.g., speech recognition API). The input is the decoded and decompressed digital audio data, and the output is text data such as "What's the weather like today?"

[1827] Step 6: Natural Language Processing (Server)

[1828] The server analyzes the text data obtained by speech recognition using a natural language processing engine (e.g., a natural language processing library) to understand the user's intent. The input is text data, and the output is the analyzed user's intent (e.g., getting weather information).

[1829] Step 7: Information Acquisition (Server)

[1830] The server retrieves the necessary information (e.g., weather information) from an external service based on the analyzed user intent. For example, it calls a weather forecast API to retrieve current weather information. The input is the user intent, and the output is the weather information retrieved from the external service (e.g., sunny, temperature 25 degrees).

[1831] Step 8: Response Generation (Server)

[1832] The server uses a generative AI model to generate a natural language response message based on the acquired information. For example, it generates a message such as "It's sunny today, and the temperature is 25 degrees." The input is weather information, and the output is the text data of the response message.

[1833] Step 9: Generate audio data (server)

[1834] The server converts the generated response message into audio data using a text-to-speech engine, and encodes and compresses it. The input is the text data of the response message, and the output is the compressed and encoded audio data.

[1835] Step 10: Sending the Response (Server)

[1836] The server sends the generated audio data to the device via the network. The input is the compressed and encoded audio data, and the output is the transmission of an HTTP response to the device.

[1837] Step 11: Receive and Play Response (Terminal)

[1838] The device receives the audio data sent from the server, decodes and decompresses it. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data. It then plays the audio data through the device's speaker and provides the user with a voice response saying, "Today is sunny and the temperature is 25 degrees."

[1839] (Application example 1)

[1840] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1841] Traditionally, customer service in physical stores has relied on a limited number of staff, making it difficult to provide fast and accurate information, especially during busy times. Furthermore, when customers wanted information about a product, they had to search for it themselves, which took time and effort. This could lead to a decline in customer satisfaction. Therefore, there is a demand for a system that allows customers to easily obtain product information through voice input.

[1842] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1843] In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intent, means for acquiring information from an external service based on the user's intent, means for generating a response message in natural language from the acquired information, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, and the system is used to provide product information in a physical store, means for acquiring information from a product database based on the user's intent, and means for generating and playing the acquired information as a response message. This enables customers to quickly and accurately obtain product information through voice input.

[1844] "Voice input" is a means by which a user provides instructions or information to a system using voice.

[1845] "Digital data" refers to data that has been converted from analog audio information into a digital format.

[1846] A "server" is a computer system that provides various services over a network.

[1847] A "voice recognition engine" is software that analyzes input voice data and converts it into text data.

[1848] "Natural language processing means" refers to techniques and methods for analyzing text data and understanding user intent.

[1849] "External services" are other systems or APIs that the server uses in response to user requests.

[1850] A "response message" is a text response generated based on the user's intent.

[1851] A "text-to-speech engine" is software for converting text data into audio data.

[1852] A "terminal" is a hardware device that is directly operated by a user and receives and plays audio.

[1853] A "product database" is a collection of data that collects and stores product information from physical stores.

[1854] A "physical store" is a store that sells products at a physical location.

[1855] "Customers" refer to typical consumers who use physical stores.

[1856] System Overview

[1857] This invention relates to a system that enables customers in a physical store to quickly and accurately obtain product information through voice input. The system is primarily composed of hardware and software for receiving voice input, converting it to digital data, transmitting it to a server, performing voice recognition, natural language processing, information acquisition, response generation, and voice output.

[1858] Hardware and software used

[1859] Terminal: Smartphone or smart device (including microphone, processor, and network communication function)

[1860] Server: Speech recognition engine, natural language processing engine, text-to-speech engine

[1861] External service: Product database access API

[1862] Operating procedure

[1863] Usage example

[1864] Consider a scenario in which a customer uses a smartphone in a physical store to ask about product information. The customer speaks into the smartphone's microphone, saying, "Tell me the price of this product."

[1865] 1. Receiving voice input (terminal): The user speaks into the smartphone's microphone. The terminal converts this voice into digital data, compresses it, and encodes it.

[1866] 2. Sending digital data (terminal): The terminal sends the digital data to the server via network communication.

[1867] 3. Speech Recognition (Server): The server decodes and decompresses the received digital data and converts it into text data using a speech recognition engine.

[1868] 4. Natural Language Processing (Server): The server analyzes the text data and recognizes the customer's request (in this case, obtaining the product price).

[1869] 5. Information acquisition (server): The server calls the API to access the product database and acquires the price information of the relevant product.

[1870] 6. Response generation (server): Based on the obtained price information, a response message such as "The price of this product is XX yen" is generated.

[1871] 7. Generate voice data (Server): The server converts the response message into voice data using a text-to-speech engine, and encodes and compresses it.

[1872] 8. Sending response (server): The server sends the generated voice data to the terminal via the network.

[1873] 9. Playback of response (terminal): The terminal decodes and decompresses the received voice data and plays it back through the speaker. The customer receives a voice response such as "The price of this product is XX yen."

[1874] Specific examples

[1875] Examples of voice commands include prompts such as:

[1876] "Please tell me the price of this item."

[1877] "Tell me what products you recommend."

[1878] "What are the features of this product?"

[1879] When a customer says to their smartphone, "Tell me the price of this item," the system receives the voice input and quickly provides the product price information via voice, allowing customers to get the information they need without any hassle while in the store.

[1880] This is a specific embodiment for carrying out the invention.

[1881] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1882] Step 1:

[1883] The user provides voice input

[1884] The user speaks into the smartphone's microphone, saying, "Tell me the price of this item." The smartphone receives the voice data. In this process, the user's voice is input as an analog signal.

[1885] Step 2:

[1886] Converting audio into digital data

[1887] The terminal converts the received voice into digital data, which includes the processes of voice sampling and quantization, and outputs the analog voice data as digital data.

[1888] Step 3:

[1889] Sending digital data to a server

[1890] The terminal compresses and encodes the converted digital data and sends it to the server via network communication. The input is the converted digital data and the output is the compressed data sent to the server.

[1891] Step 4:

[1892] Converting audio data into text data

[1893] The server decodes and decompresses the received data and converts the voice data into text data using a speech recognition engine. The input is compressed voice data and the output is text data.

[1894] Step 5:

[1895] Analyzing text data to understand user intent

[1896] The server analyzes the text data and uses a natural language processing engine to understand the user's intent. In this process, the input is the text data to be analyzed, and the output is the intent data, such as "I want to know the price of this product."

[1897] Step 6:

[1898] Get product information

[1899] Based on the user's intent, the server calls the API to access the product database and obtains the price information of the relevant product. The input is the understood user intent, and the output is the obtained price information.

[1900] Step 7:

[1901] Generate a response message

[1902] The server generates a response message in natural language based on the acquired product information. The input for this process is product price information, and the output is a response message saying, "The price of this product is XX yen."

[1903] Step 8:

[1904] Converting response messages into audio data

[1905] The server converts the generated response message into voice data using a text-to-speech engine. The input is the response message, and the output is the voice data.

[1906] Step 9:

[1907] Sending audio data to the device

[1908] The server compresses and encodes the converted audio data and sends it to the terminal via network communication. The input is the generated audio data, and the output is the compressed data sent to the terminal.

[1909] Step 10:

[1910] Playing audio data

[1911] The terminal decodes and decompresses the received audio data and plays it through a speaker. The input is compressed audio data and the output is audio information played back to the user.

[1912] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1913] System Overview

[1914] This invention relates to an AI butler system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses accordingly. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[1915] System configuration

[1916] This system mainly consists of a terminal and a server.

[1917] Terminal

[1918] The device receives voice input from the user and records, converts, compresses, and encodes it. It includes a microphone, a processor, and network communication functions. The device transmits voice data to the server and receives and plays response voice data from the server.

[1919] server

[1920] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. It also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server obtains necessary information from external services and generates a response message in natural language. It also has an emotion recognition engine that analyzes the user's emotions and adjusts the response based on the emotions.

[1921] System Operation

[1922] The operation of the system of the present invention will now be described in detail.

[1923] Brief Description

[1924] 1. Receiving voice input (terminal)

[1925] A user speaks a voice command into the device, such as "What's the weather today?" The device's microphone converts this voice into digital data, which is then compressed and encoded.

[1926] 2. Sending audio data (terminal)

[1927] The digital data is transmitted over a network to a server.

[1928] 3. Speech recognition (server)

[1929] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine, which generates the text "What's the weather like today?"

[1930] 4. Emotion Recognition (Server)

[1931] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, detecting, for example, whether the user is tired or happy.

[1932] 5. Text Analysis and Natural Language Processing (Server)

[1933] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[1934] 6. Information Acquisition (Server)

[1935] The server calls an external weather forecast API to obtain current weather information according to the user's request.

[1936] 7. Generating a Response Message (Server)

[1937] The server generates a natural-sounding response message based on the weather information it has acquired, e.g., "It's sunny today, and the temperature is 25 degrees." It also adjusts the message to an appropriate tone, taking into account the user's emotions.

[1938] 8. Text-to-speech conversion (server)

[1939] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[1940] 9. Sending audio data (server)

[1941] The server transmits the generated voice data to the terminal via the network.

[1942] 10. Response playback (terminal)

[1943] The device decodes and decompresses the received audio data and plays it through the speaker.

[1944] Specific examples

[1945] The operation of this system will be explained below using specific scenarios.

[1946] Example: Obtaining weather information based on emotions

[1947] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[1948] 1. The device records the audio and converts it into digital data.

[1949] 2. Send the digital data to the server.

[1950] 3. The server converts the voice data into text and recognizes it as "What's the weather like today?"

[1951] 4. The server uses an emotion recognition engine to determine that the user is tired.

[1952] 5. The server understands that you are requesting weather information and retrieves the latest weather information from the weather forecast API.

[1953] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[1954] 7. The response message is converted into voice data and sent to the terminal.

[1955] 8. The device plays the audio data and tells the user, "It's sunny today and the temperature is 25 degrees. Good luck."

[1956] In this way, the system can take into account the user's emotions and provide more personalized responses, creating a more natural and comfortable interface for users, enabling them to efficiently search for complex information and perform tasks.

[1957] The processing flow will be explained below.

[1958] Step 1:

[1959] Receiving voice input (user, device)

[1960] The user speaks to the device, saying, "Tell me the weather today." The device's microphone converts this voice into digital data, capturing the voice data.

[1961] Step 2:

[1962] Audio data buffering (terminal)

[1963] The terminal temporarily stores the received digital audio data in a memory buffer and divides the data into chunks of a certain size.

[1964] Step 3:

[1965] Audio data compression and encoding (terminal)

[1966] The terminal compresses the digital audio data in the buffer and encodes it into a format suitable for network transmission.

[1967] Step 4:

[1968] Sending audio data (terminal)

[1969] The device sends the compressed and encoded audio data to the server via the network.

[1970] Step 5:

[1971] Receiving an HTTP request (server)

[1972] The server receives the HTTP request sent from the terminal and acquires the audio data.

[1973] Step 6:

[1974] Decoding and decompression (server)

[1975] The server decodes the received audio data and decompresses the compressed data into the original digital audio data.

[1976] Step 7:

[1977] Speech recognition (server)

[1978] The server uses a speech recognition engine to analyze the digital voice data and convert it into text data, where the speech recognition engine generates the text "What's the weather like today?"

[1979] Step 8:

[1980] Emotion recognition (server)

[1981] The server uses an emotion recognition engine to analyze the user's emotions from the voice data, for example, detecting whether the user is tired or happy from the tone of their voice and speaking patterns.

[1982] Step 9:

[1983] Text analysis and natural language processing (server)

[1984] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. In this case, it determines that the request is for weather information.

[1985] Step 10:

[1986] Information acquisition (server)

[1987] In response to a user request, the server calls the weather forecast API of an external service to obtain current weather information.

[1988] Step 11:

[1989] Generate a response message (server)

[1990] The server generates a natural-sounding response message based on the acquired weather information. For example, the generated message might be, "It's sunny today, and the temperature is 25 degrees." In addition, it adds encouraging words such as "Keep up the good work!" based on the user's emotions (e.g., tiredness) analyzed by an emotion recognition engine.

[1991] Step 12:

[1992] Text-to-speech conversion (server)

[1993] The server uses a text-to-speech engine to convert the generated text response message into audio data.

[1994] Step 13:

[1995] Audio data encoding and compression (server)

[1996] The server compresses and encodes the generated audio data into an efficient transmission format.

[1997] Step 14:

[1998] Sending a response (server)

[1999] The server sends the encoded and compressed audio data to the terminal as an HTTP response.

[2000] Step 15:

[2001] Decoding and decompression (terminal)

[2002] The terminal decodes and decompresses the received encoded and compressed audio data to restore it to its original form.

[2003] Step 16:

[2004] Audio playback (device)

[2005] The device then plays the decoded audio data through the speaker and communicates the response to the user, specifically, "It's sunny today, and the temperature is 25 degrees. Good luck!"

[2006] Example 2

[2007] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2008] Conventional speech recognition systems simply convert a user's speech into text and understand their intent, but it is difficult to generate responses that take the user's emotions into account. As a result, many responses are mechanical and ignore the user's emotional state, resulting in a limited user experience. Furthermore, the entire process from voice input to response generation requires immediacy and accuracy of information, but there has been a lack of technology to achieve this.

[2009] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2010] In this invention, the server includes means including a voice recognition engine for converting voice data into text data, natural language processing and emotion recognition means for analyzing the text data and understanding the user's intentions and emotions, and means for acquiring information from an external database, thereby enabling the server to provide a more personalized response to the user by converting the user's voice input into digital data and analyzing the user's emotions as well as their intentions to generate an appropriate response message.

[2011] "Voice input" refers to an input format in which a user gives instructions or asks questions using voice.

[2012] "Digital data" refers to data obtained by converting analog audio signals into digital format.

[2013] A "server" is a computer system that communicates with other computers and devices over a network and provides specific functions or services.

[2014] A "speech recognition engine" is a piece of software or hardware that converts voice data into text data.

[2015] "Natural language processing" is a technology that enables computers to understand, analyze, and generate human language.

[2016] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data and text data.

[2017] An "external database" is a collection of information that exists outside the system and is a database that is accessed to obtain required information.

[2018] "Text data" is character string data converted from voice input by a voice recognition engine.

[2019] A "response message" is a reply in natural language format that the system generates in response to a user's question or instruction.

[2020] A "text-to-speech engine" is a piece of software or hardware for converting text data into speech data.

[2021] A "terminal" is a device that allows a user to input voice and has the function of communicating with a server.

[2022] "Decoding" is the process of returning encoded data to its original form.

[2023] "Decompression" is the process of restoring compressed data to its original state.

[2024] System Overview

[2025] This invention relates to an AI system that receives a user's voice input, analyzes it, and generates an appropriate response. It is particularly characterized by its ability to recognize the user's emotions and adjust responses based on those emotions. This system effectively performs speech recognition, natural language processing, emotion recognition, information acquisition, response generation, and voice output.

[2026] System configuration

[2027] This system mainly consists of a terminal and a server.

[2028] Terminal

[2029] The device receives voice input from the user and records, converts, compresses, and encodes it. This includes a microphone, a processor, and network communication capabilities. Specifically, the device's microphone (e.g., a microphone) converts the voice into digital data, and the processor (e.g., a central processing unit) compresses and encodes the data. The device then uses its network communication capabilities (e.g., a wireless communication module) to send the data to a server, receive response voice data from the server, and play it back.

[2030] server

[2031] The server includes a speech recognition engine that receives voice data sent from the terminal and converts it into text data. For example, the server converts the received voice data into text data using a speech recognition engine (e.g., generic voice recognition software). The server also has a natural language processing means that analyzes the text data and understands the user's intent. Based on the user's intent, the server retrieves necessary information from an external database and generates a natural language response message. The server also has an emotion recognition engine (e.g., generic emotion analysis software) that adjusts the response based on the user's emotion.

[2032] System Operation

[2033] Receiving and processing audio input (terminal)

[2034] When a user speaks a voice command into the device, such as "What's the weather today?", the device's microphone converts the voice into digital data, which is then compressed and encoded by a processor and sent to a server via network communications.

[2035] Speech and emotion recognition (server)

[2036] The server decodes and decompresses the received voice data and converts it into text data using a speech recognition engine. For example, text data such as "What's the weather like today?" is generated. Then, an emotion recognition engine is used to analyze the user's emotions from the voice data. For example, emotions such as tiredness or happiness can be detected.

[2037] Text analysis and information retrieval (server)

[2038] The server uses a natural language processing engine to analyze the text data obtained by speech recognition and understand the user's intent. For example, it determines that the user is "requesting weather information." The server then calls an external database (e.g., a general-name weather forecast database) to obtain current weather information.

[2039] Generate and send a response message (server)

[2040] The server generates a natural-sounding response message based on the acquired weather information. For example, it might generate a message like "It's sunny today, and the temperature is 25 degrees." Taking the user's emotions into consideration, the server might add words of encouragement, such as "Good luck." The server then converts the generated text response message into audio data using a text-to-speech engine and sends it to the device.

[2041] Response playback (terminal)

[2042] The device decodes and decompresses the received audio data and plays it back through a speaker (e.g., a general-purpose audio output device). Specifically, it conveys to the user something like, "Today is sunny, and the temperature is 25 degrees. Good luck."

[2043] Specific examples

[2044] The operation of this system will be explained below using specific scenarios.

[2045] Example: Obtaining weather information based on emotions

[2046] A user speaks to a smart speaker in the living room in the morning, asking, "What's the weather today?" The user sounds a little tired.

[2047] 1. The device records the sound and converts it into digital data (microphone: the common name for a microphone).

[2048] 2. Send the digital data to the server (network communication: commonly known as a wireless communication module).

[2049] 3. The server converts the voice data into text (voice recognition engine: generic name for voice recognition software) and recognizes the text as "Tell me what the weather is today."

[2050] 4. The server uses an emotion recognition engine (commonly known as emotion analysis software) to determine that the user is tired.

[2051] 5. The server understands that you are requesting weather information (natural language processing engine: a general term for natural language processing software) and retrieves the latest weather information from a weather forecast database (a general term for an external database).

[2052] 6. The server generates a response message that takes the user's emotions into consideration, such as, "It's sunny today and the temperature is 25 degrees. Good luck."

[2053] 7. The response message is converted into voice data (text-to-speech engine: a common name for speech synthesis software) and sent to the terminal.

[2054] 8. The device plays audio data (speaker: a common name for an audio output device) and tells the user, "It's sunny today, and the temperature is 25 degrees. Good luck."

[2055] Prompt Sentence Examples

[2056] "Tell me about today's weather."

[2057] "Check tomorrow's schedule."

[2058] "Play some uplifting music"

[2059] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2060] Step 1:

[2061] The device receives voice input from the user. When the user says to the device, "What's the weather today?", the microphone converts the voice into digital data. The input is analog voice and the output is digital data. This data is then processed by a processor within the device.

[2062] Step 2:

[2063] The device compresses and encodes the digital data. Specifically, a processor (e.g., a central processing unit) compresses the digital data into MP3 or AAC format and encodes the data into a format suitable for the network. The input is digital audio data, and the output is compressed and encoded audio data.

[2064] Step 3:

[2065] The device sends the compressed and encoded audio data to the server via a network communication function (e.g., a wireless communication module). The data is encrypted to ensure secure communication. The input is the compressed and encoded audio data, and the output is the transmission of the audio data to the server.

[2066] Step 4:

[2067] The server decodes and decompresses the received audio data, specifically using the server's (e.g., computer system's) decoding and decompression capabilities (e.g., generic software) to return the audio data to its original digital form. The input is the compressed and encoded audio data, and the output is the decoded and decompressed audio data.

[2068] Step 5:

[2069] The server uses a speech recognition engine (e.g., speech recognition software) to convert the decoded and decompressed audio data into text data. Specifically, the speech recognition engine analyzes the audio data and generates the text "What's the weather like today?" The input is the decoded audio data, and the output is text data.

[2070] Step 6:

[2071] The server uses an emotion recognition engine (e.g., emotion analysis software) to analyze the user's emotions from the voice data. Specifically, the emotion recognition engine analyzes the tone and pitch of the voice to identify the user's emotions, such as tiredness or joy. The input is the voice data, and the output is the user's emotional information as a result of the analysis.

[2072] Step 7:

[2073] The server uses a natural language processing engine (e.g., natural language processing software) to analyze the text data and understand the user's intent. Specifically, the natural language processing engine processes the text data and determines that it is a "request for weather information." The input is the text data and the user's emotional information, and the output is the analysis result of the user's intent.

[2074] Step 8:

[2075] The server, according to the user's request, calls an external database (e.g., a weather forecast database) to obtain current weather information. Specifically, the server sends a request to the database via an API to obtain weather information. The input is the user's intent and the API request from the external database, and the output is weather information.

[2076] Step 9:

[2077] The server generates a natural response message based on the acquired weather information. Specifically, the server uses its response message generation function to generate a message such as "It's sunny today, and the temperature is 25 degrees. Good luck!" The input is weather information and the user's emotional information, and the output is the response message.

[2078] Step 10:

[2079] The server uses a text-to-speech engine (e.g., speech synthesis software) to convert the generated text response message into audio data. Specifically, the text-to-speech engine converts the text data into audio data. The input is the response message, and the output is audio data.

[2080] Step 11:

[2081] The server sends the generated voice data to the device via the network communication function. The data is again encrypted to ensure secure communication. The input is voice data, and the output is the transmission of voice data to the device.

[2082] Step 12:

[2083] The device decodes and decompresses the received voice data and plays it back through the speaker. Specifically, the device uses its decoding function to return the voice data to its original format and plays back from the speaker, "It's sunny today, and the temperature is 25 degrees. Good luck." The input is the voice data received from the server, and the output is a voice response to the user.

[2084] (Application example 2)

[2085] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2086] To improve passenger comfort in autonomous vehicles, systems that can accurately understand passenger emotions and intentions and provide appropriate responses based on those emotions are required. However, current voice assistant systems for autonomous vehicles lack emotion recognition capabilities and are unable to provide personalized responses based on passenger emotions, which can cause discomfort to passengers.

[2087] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into digital data, means for transmitting the digital data to the server, means including a voice recognition engine for converting the digital data into text data, natural language processing means for analyzing the text data and understanding the user's intention, means for acquiring information from an external service based on the user's intention, means for generating the acquired information into a response message in natural language, means including a text-to-speech engine for converting the response message into voice data, means for transmitting the voice data to a terminal, means for playing the voice data, means for analyzing the user's emotion from the voice data, and means for adjusting the response message based on the user's emotion. This enables passengers in an autonomous vehicle to receive a personalized response based on their emotions.

[2088] The "means for receiving user voice input" is a device or function that receives voice commands from the user through a microphone.

[2089] The "means for converting the voice input into digital data" refers to a device or function that converts analog input voice data into digital data.

[2090] The "means for transmitting the digital data to the server" refers to a device or function that transmits the digital data to the server via network communication.

[2091] The "means including a voice recognition engine for converting said digital data into text data" refers to a device or function that converts digital voice data into text form using voice recognition technology.

[2092] The "natural language processing means for analyzing the text data and understanding the user's intent" is a device or function that uses natural language processing technology to analyze the text data and understand the user's intent.

[2093] The "means for acquiring information from an external service based on the user's intention" is a device or function for acquiring necessary information from an external information providing service based on the analyzed user's intention.

[2094] The "means for generating a response message in a natural language from the acquired information" refers to a device or function that generates a response message in a natural language based on the acquired information.

[2095] The "means including a text-to-speech engine for converting the response message into voice data" refers to a device or function that converts the generated text message into voice data.

[2096] The "means for transmitting the voice data to the terminal" is a device or function for transmitting the generated voice data to the terminal.

[2097] The "means for reproducing the audio data" refers to a device or function such as a speaker that reproduces audio data on the terminal.

[2098] The "means for analyzing the user's emotions from the voice data" refers to a device or function that analyzes the voice data and identifies the user's emotional state.

[2099] The "means for adjusting the response message based on the user's emotions" refers to a device or function that adjusts the content and tone of the response message based on the analyzed emotional state of the user.

[2100] A "means for receiving voice input from a passenger of an automated vehicle" is a device or function within the automated vehicle that receives voice input from a passenger.

[2101] A "generative AI model" refers to a trained artificial intelligence model used for natural language processing and intent analysis.

[2102] A "prompt sentence" is an input sentence containing instructions or questions from a user given to a generative AI model.

[2103]

[2104] This invention relates to a voice assistant system for improving passenger comfort in autonomous vehicles. The system can provide personalized services to passengers by using AI technology to receive user voice input, analyze it, and generate appropriate responses.

[2105] System Configuration

[2106] This system mainly consists of a terminal and a server.

[2107] Terminal

[2108] The terminal is installed inside the autonomous vehicle and has the following functions:

[2109] 1. Audio input reception:

[2110] Passenger voice commands are received via a microphone.

[2111] 2. Audio digital conversion:

[2112] Converts audio input into digital data.

[2113] 3. Data transmission:

[2114] The converted digital data is sent to a server.

[2115] 4. Audio data playback:

[2116] The response voice data sent from the server is played back.

[2117] server

[2118] The server receives the digital audio data sent from the terminal and provides the following functions:

[2119] 1. Speech Recognition Engine:

[2120] Converts digital voice data into text data using the Google Cloud Speech-to-Text API or similar.

[2121] 2. Natural Language Processing Engine:

[2122] The converted text data is analyzed to understand the user's intent. OpenAI's generative AI models (such as GPT-3 / 4) are used.

[2123] 3. Emotion Recognition Engine:

[2124] Analyzes user emotions from voice data, using Azure Emotion API etc.

[2125] 4. Information acquisition means:

[2126] Obtain necessary information from external services based on the user's intent.

[2127] 5. Response message generation:

[2128] A natural language response message is generated based on the acquired information and the results of sentiment analysis.

[2129] 6. Text-to-speech engine:

[2130] The generated text message is converted into audio data, using the Google Text-to-Speech API or similar.

[2131] 7. Means of data transmission:

[2132] The converted audio data is sent to the terminal.

[2133] System Operation

[2134] The system works as follows:

[2135] 1. Receiving voice input:

[2136] The user (passenger) issues a voice command to the terminal, for example, a prompt such as "What's the weather like today?"

[2137] 2. Data transmission:

[2138] The terminal transmits the voice data to the server.

[2139] 3. Speech Recognition and Natural Language Processing:

[2140] The server converts the received voice data into text and analyzes it using a natural language processing engine to understand the user's intent.

[2141] 4. Emotion recognition:

[2142] An emotion recognition engine is used to analyze the user's emotions and tailor the response message based on the emotions.

[2143] 5. Information Acquisition and Response Generation:

[2144] External information is acquired based on the user's intention, and a personalized response message is generated.

[2145] 6. Audio data conversion and transmission:

[2146] The text-to-speech engine converts the response message into voice data, which is then sent to the terminal.

[2147] 7. Response playback:

[2148] The terminal plays back the transmitted voice data and conveys the response to the passenger.

[2149] Specific examples

[2150] For example, if a passenger asks the terminal, "What's the weather like today?", and the passenger sounds a little tired, the system works as follows:

[2151] 1. The device receives the audio, converts it into digital data, and sends it to the server.

[2152] 2. The server uses a speech recognition engine to convert the text into "What's the weather like today?"

[2153] 3. Use an emotion recognition engine to detect tired emotions from voice data.

[2154] 4. The natural language processing engine analyzes the user's intent and understands that they are looking for weather information.

[2155] 5. The server retrieves the latest weather information from the weather forecast API.

[2156] 6. Based on the emotion recognition results, generate a response message such as "It's a sunny day today. Please relax."

[2157] 7. The response message is converted into voice data and sent to the terminal.

[2158] 8. The device plays audio data telling the passenger, "It's a sunny day today. Please relax."

[2159] An example prompt is:

[2160] "Please tell me the weather today."

[2161] "Please tell me the current traffic information."

[2162] This enables the system to provide passengers with personalized services and a more comfortable travel experience.

[2163] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2164] Step 1: Receiving audio input

[2165] The terminal receives voice commands from the user (passenger) using a microphone. The input is the passenger's voice, and the output is analog voice data. Specifically, the microphone captures the voice and converts it into a digital signal.

[2166] Step 2: Sending audio data

[2167] The terminal transmits the converted digital audio data to the server. The input is the digital audio data, and the output is the data transmitted to the server. Specifically, the terminal transmits the data to the server via network communication.

[2168] Step 3: Voice Recognition

[2169] The server converts the received digital voice data into text data using a speech recognition engine. The input is digital voice data, and the output is text data. Specifically, the server calls the Google Cloud Speech-to-Text API to obtain the text data.

[2170] Step 4: Natural Language Processing

[2171] The server analyzes the text data using a natural language processing engine to understand the user's intent. The input is text data, and the output is the analyzed intent. Specifically, the server uses OpenAI's generative AI model (GPT-3 / 4) to analyze the text data and determine the intent.

[2172] Step 5: Emotion Recognition

[2173] The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The input is the voice data, and the output is the user's emotional state. Specifically, the server calls the Azure Emotion API to analyze the voice data and identify the emotion.

[2174] Step 6: Information Acquisition

[2175] The server retrieves the necessary information from external services based on the analyzed intent. The input is the user's intent, and the output is the retrieved information. Specifically, the server retrieves information by calling external weather forecast APIs and traffic information APIs.

[2176] Step 7: Generate a response message

[2177] The server generates a response message in natural language based on the acquired information and the results of emotion analysis. The input is the acquired information and the user's emotional state, and the output is the response message. Specifically, the server uses the generative AI model to create a text message.

[2178] Step 8: Text-to-speech

[2179] The server converts the generated text message into audio data. The input is the response message, and the output is audio data. Specifically, the server calls the Google Text-to-Speech API to convert the text into audio data.

[2180] Step 9: Sending audio data

[2181] The server transmits the converted voice data to the terminal. The input is the voice data, and the output is the data transmitted to the terminal. Specifically, the server transmits the voice data to the terminal via network communication.

[2182] Step 10: Playing back the response

[2183] The terminal plays the received voice data. The input is voice data, and the output is the voice heard by the passenger. Specifically, the voice data is played through the terminal's speaker to convey the response to the passenger.

[2184] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2185] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2186] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2187] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2188] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2189] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2190] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2191] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various bal...

Claims

1. means for receiving a user's voice input; means for converting said voice input into digital data; means for transmitting the digital data to a server; means including a voice recognition engine for converting the digital data into text data; natural language processing means for analyzing the text data and understanding the user's intent; means for acquiring information from an external service based on the user's intention; means for generating the acquired information into a response message in natural language; means including a text-to-speech engine for converting the response message into audio data; means for transmitting the voice data to a terminal; a system including means for playing said audio data;

2. 10. The system of claim 1, further comprising means for initiating said receiving of voice input by triggering a user voice command.

3. 2. The system of claim 1, further comprising: means for compressing and encoding said audio data; and means for decoding and decompressing said audio data.

Images