System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an audio input from a user; means for transmitting the audio input to a server; means for converting the audio input into text data; means for generating an appropriate response based on the text data; means for converting the response into audio data; means for transmitting the audio data to the user; and means for playing the audio data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional voice assistant systems are cumbersome because they require users to manually input information into smartphones or other devices. Furthermore, voice responses are often displayed on the device's screen, requiring users to visually confirm the response, which can interrupt other tasks. This situation is inconvenient, especially when driving or when hands are full, and can reduce a user's work efficiency. The present invention aims to solve these problems and provide a more seamless and efficient voice interaction. [Means for solving the problem]

[0005] The present invention provides a terminal equipped with means for capturing voice input from a user and transmitting the voice input to a server. The server includes means for converting the voice input into text data and means for generating an appropriate response based on the text data. The server further includes means for converting the generated text response into voice data and transmitting the voice data to the user. The user can play the voice data on the terminal. Furthermore, a natural language processing engine is used to analyze the voice input, enabling the generation of appropriate and sophisticated responses. This allows the user to ask questions or give instructions by voice and hear the responses by voice without manually operating the device, thereby avoiding interruption to other tasks.

[0006] "Voice input" refers to the voice spoken by a user, which is captured using a device such as a microphone.

[0007] "Capture" means obtaining data such as audio or images using a specific device or means.

[0008] "Server" means a computer system that processes and stores data over a network.

[0009] "Text data" refers to character string information obtained by analyzing and converting input data such as voice.

[0010] "Response" means a reply or instruction generated by a system in response to input from a user.

[0011] "Audio data" refers to an audio signal converted from text data using speech synthesis technology.

[0012] "Playback" means outputting digital data in a format that humans can intuitively understand (for example, audio or images).

[0013] "Natural language processing engine" means software or a system that parses user input in natural language and generates an appropriate response.

[0014] "Analysis" means breaking down and interpreting data to understand its meaning and structure. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. Specific embodiments of this system are described below.

[0037] Overall system configuration

[0038] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[0039] Detailed program processing

[0040] Server Processing

[0041] 1. Receiving voice input

[0042] The server receives the voice data sent by the user and passes it to the voice recognition API.

[0043] 2. Converting voice data to text

[0044] The server converts the voice data into text data using a voice recognition API, which performs, for example, spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[0045] 3. Natural Language Processing and Response Generation

[0046] The server then inputs the resulting text data into a natural language processing engine, which analyzes the voice commands and questions and generates an appropriate response. For example, if a user asks for the weather forecast, the engine queries a weather forecast API and generates a response based on the results.

[0047] 4. Generating Audio Data

[0048] The server then passes the generated text response to a speech synthesis API, which converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[0049] 5. Sending audio data

[0050] The server transmits the generated voice data to the user's terminal via the network.

[0051] Terminal handling

[0052] 1. Capture voice input

[0053] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[0054] 2. Receiving and playing audio data

[0055] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[0056] Specific examples

[0057] Example 1: Checking the schedule

[0058] 1. User: "What's my schedule for tomorrow?"

[0059] 2. Device: Uses microphone to capture audio and send it to the server.

[0060] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[0061] 4. Server: The natural language processing engine calls the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[0062] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0063] 6. Server: Sends audio data to the device.

[0064] 7. Terminal: Play audio data.

[0065] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0066] Example 2: Checking the weather forecast

[0067] 1. User: "What's the weather like today?"

[0068] 2. Device: Uses microphone to capture audio and send it to the server.

[0069] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[0070] 4. Server: The natural language processing engine calls the weather forecast API and generates a response: "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[0071] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0072] 6. Server: Sends audio data to the device.

[0073] 7. Terminal: Play audio data.

[0074] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0075] This system allows users to efficiently obtain information by voice without manual input or visual confirmation, providing great convenience especially when hands are busy or during tasks that require visual concentration.

[0076] The processing flow will be explained below.

[0077] Step 1:

[0078] The user inputs voice into the Bluetooth earphones or audio glasses. The user's speech is a question or instruction such as "Tell me today's weather."

[0079] Step 2:

[0080] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[0081] Step 3:

[0082] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[0083] Step 4:

[0084] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[0085] Step 5:

[0086] The server passes the text data to a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. For example, it may determine that the request is for a weather forecast. The natural language processing engine analyzes the input text data and determines an appropriate response.

[0087] Step 6:

[0088] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, the server generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0089] Step 7:

[0090] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[0091] Step 8:

[0092] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the encoded voice data to the terminal via the network.

[0093] Step 9:

[0094] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[0095] Step 10:

[0096] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0097] This process flow allows users to obtain information and execute instructions using only their voice, without having to manually operate the device. This allows users to solve problems efficiently without interrupting other tasks.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] Conventional voice interface systems often require users to manually input information or look at a screen when obtaining it through voice, limiting their convenience. Furthermore, these systems are only effective in specific locations and situations, making flexible use difficult. Furthermore, if the accuracy of speech recognition and response generation is low, users may not receive satisfactory service.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes means for acquiring voice input from a user, means for transmitting the voice input to a processing device, means for converting the voice input into text data, means for generating an appropriate response based on the text data, means for converting the response into voice data, means for transmitting the voice data to the user, and means for playing back the voice data. This allows the user to efficiently acquire information in a natural, interactive format without having to manually input or check a screen. Specifically, by using a voice recognition API and a natural language processing engine, the server can analyze the user's intent with high accuracy and generate and provide an appropriate response.

[0103] A "means for obtaining voice input" is a device that senses a user's speech and converts it into a voice signal.

[0104] The "means for transmitting to a processing unit" is a device or function that transmits the captured audio signal as digital data to a central processing unit via a network.

[0105] "Means for converting into text data" refers to a function or device that analyzes a voice signal and converts it into text data.

[0106] The "means for generating an appropriate response" refers to a function or device that analyzes the user's intention based on character data and generates a response in natural language.

[0107] The "means for converting into voice data" refers to a function or device that converts the generated text response into voice data.

[0108] The "means for transmitting to the user" refers to a function or device that transmits voice data to the user's terminal via a network.

[0109] The "means for reproducing audio data" is a device such as a speaker that reproduces audio data received by the user's terminal.

[0110] The "means for analyzing in natural language" refers to a function or device for analyzing acquired text data and understanding the intent of the user's questions and instructions.

[0111] An "external language processing engine" is an engine with natural language processing capabilities that is provided through a cloud service or external API.

[0112] This invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connection, and receive responses by voice. Here we will show how to specifically implement this system.

[0113] Overall system configuration

[0114] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[0115] Hardware and software used

[0116] Devices: Bluetooth earphones, audio glasses with built-in mobile connection

[0117] Server: High-performance cloud computing services

[0118] Speech recognition API: Google® Speech-to-Text, etc.

[0119] Natural language processing engine: OpenAI (registered trademark), GPT-3 (registered trademark), etc.

[0120] Speech synthesis API: Google Text-to-Speech, etc.

[0121] Network: Internet connection (Wi-Fi, mobile data)

[0122] Processing of various data

[0123] Receiving and processing voice input

[0124] User: Ask questions and give instructions using voice, just like in everyday conversation.

[0125] The device captures the user's voice using a built-in microphone, and the voice data is compressed through digital signal processing.

[0126] The terminal transmits the compressed audio data to the server via the network.

[0127] Speech-to-text transcription and analysis

[0128] The server receives the audio data over the network.

[0129] The server converts the received voice data into text using the Google Speech-to-Text API.

[0130] The server passes the converted text data to a natural language processing engine such as OpenAI GPT-3, which analyzes the user's intent and generates an appropriate response.

[0131] Response generation and speech synthesis

[0132] The server passes the generated text response to a speech synthesis API, such as the Google Text-to-Speech API, and converts it into audio data.

[0133] The server encodes the generated voice data and transmits it to the user's terminal.

[0134] Playing audio

[0135] The terminal decodes the received audio data and plays it on the speaker.

[0136] The user hears the response from the server through the played audio.

[0137] Specific examples

[0138] Example 1: Checking the schedule

[0139] 1. User: "What's my schedule for tomorrow?"

[0140] 2. Device: Uses microphone to capture audio and send it to the server.

[0141] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[0142] 4. Server: The natural language processing engine queries the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[0143] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0144] 6. Server: Sends audio data to the device.

[0145] 7. Terminal: Play audio data.

[0146] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0147] Example 2: Checking the weather forecast

[0148] 1. User: "What's the weather like today?"

[0149] 2. Device: Uses microphone to capture audio and send it to the server.

[0150] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[0151] 4. Server: The natural language processing engine queries the weather API and generates a response: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[0152] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0153] 6. Server: Sends audio data to the device.

[0154] 7. Terminal: Play audio data.

[0155] 8. User: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[0156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0157] Step 1:

[0158] The user asks questions or gives instructions by voice.

[0159] Input: User utterance "What's the weather like today?"

[0160] Specific actions: The user speaks through Bluetooth earphones or audio glasses.

[0161] Step 2:

[0162] The terminal captures the user's speech.

[0163] Input: User's spoken utterance

[0164] What it does: It uses a built-in microphone to capture the user's speech as digital audio data.

[0165] Output: Digital audio data

[0166] Step 3:

[0167] The terminal transmits the captured audio data to the server.

[0168] Input: Digital audio data

[0169] What it does: It compresses digital audio data and sends it over the network to a server.

[0170] Output: Compressed audio data

[0171] Step 4:

[0172] The server passes the received voice data to the voice recognition API.

[0173] Input: Compressed audio data

[0174] Specific operation: Decodes the audio data and passes it to a speech recognition API such as the Google Speech-to-Text API.

[0175] Output: Text data "What's the weather like today?"

[0176] Step 5:

[0177] The server inputs the data converted into text by the speech recognition API into a natural language processing engine.

[0178] Input: Text data "What's the weather like today?"

[0179] What it does: Provides text data as prompts to a natural language processing engine such as OpenAI GPT-3.

[0180] Output: Response "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0181] Step 6:

[0182] The server passes the generated response text to the speech synthesis API.

[0183] Input: Response "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[0184] What it does: Passes the response text to a speech synthesis API, such as the Google Text-to-Speech API, to generate natural-sounding speech.

[0185] Output: Audio data

[0186] Step 7:

[0187] The server transmits the generated voice data to the user's terminal.

[0188] Input: Audio data

[0189] Specific operations: Encodes audio data and sends it over the network to the user's device.

[0190] Output: Audio data

[0191] Step 8:

[0192] The terminal decodes the received audio data and plays it on the speaker.

[0193] Input: Received audio data

[0194] Specific behavior: Decodes audio data and plays it to the user through the built-in speaker.

[0195] Output: Audio played: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[0196] (Application example 1)

[0197] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0198] Current voice interfaces for autonomous vehicles have difficulty accurately analyzing user voice input and providing appropriate information. Furthermore, there is a lack of technology that allows drivers to efficiently operate autonomous vehicles through voice commands, making it difficult to balance safe operation with information provision. Therefore, there is a demand for a system that can improve the accuracy of voice interfaces within autonomous vehicles and provide appropriate information needed during operation.

[0199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0200] In this invention, the server includes means for capturing voice input from a user, means for transmitting the voice input to the server, and means for converting the voice input into text data, thereby enabling generation of appropriate route guidance based on position information within the autonomous mobile body.

[0201] "Voice input" is voice data uttered by the user through a microphone.

[0202] A "server" is a computer system that analyzes voice and text data over a network and generates an appropriate response.

[0203] "Text data" is character information converted by speech recognition.

[0204] A "response" is information or instructions generated based on voice input.

[0205] "Audio data" is data in an audio format that is converted from text data by voice synthesis.

[0206] An "autonomous mobile object" is a machine such as a vehicle or robot that moves autonomously without human operation.

[0207] "Route guidance" refers to instructions that include route information to a destination.

[0208] "Means for analyzing in natural language" refers to methods and technologies for analyzing and understanding voice input using language processing technology.

[0209] A "map information processing engine" is a system that analyzes map data and provides route guidance and identifies the current location.

[0210] Overall system configuration

[0211] This system consists of a voice interface device installed in the autonomous vehicle and an external cloud server. Users can communicate with the autonomous vehicle through the voice interface and receive various information and instructions.

[0212] Hardware Configuration

[0213] Voice interface device for autonomous mobile body

[0214] The audio interface device includes the following hardware:

[0215] Microphone: Captures audio input.

[0216] Speaker: Plays back the generated audio data.

[0217] Communication module: Sends and receives data to and from the server.

[0218] server

[0219] It exists on the cloud and has the following functions:

[0220] Speech Recognition API: Converts voice data into text.

[0221] Natural language processing engine: Analyzes text and generates responses.

[0222] Map information processing engine: Generates route guidance based on location information.

[0223] Speech synthesis API: Converts text into audio data.

[0224] Software configuration and data processing

[0225] 1. Capture voice input

[0226] When a user speaks into the voice interface device, a built-in microphone captures the voice and transmits the data to a server.

[0227] 2. Converting voice data to text

[0228] The server converts the transmitted voice data into text data using a speech recognition API, which performs spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[0229] 3. Natural Language Processing and Response Generation

[0230] The text data is fed into a natural language processing engine, which analyzes the question or instruction and generates an appropriate response. For example, if the user requests route guidance, a map information processing engine is called to calculate the optimal route.

[0231] 4. Generate response audio data

[0232] The generated response text is passed to a speech synthesis API and converted into audio data, which takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[0233] 5. Sending and Playing Audio Data

[0234] The generated voice data is transmitted to the voice interface device through the communication module of the autonomous mobile body and played back through a speaker.

[0235] Specific examples

[0236] 1. Providing route guidance

[0237] User: "What's the quickest route to my house?"

[0238] Server: Converts speech to text using speech recognition API.

[0239] Server: The natural language processing engine analyzes the request "Tell me the shortest route to my house" and calls the map information processing engine to obtain route information.

[0240] Server: Convert the generated route guidance into voice data using a speech synthesis API.

[0241] Autonomous mobile unit: Receives and plays back audio data.

[0242] 2. Examples of prompts

[0243] "Please tell me the shortest route to my destination. Please specify the fastest route. Starting point: current location, destination: home. Please also take current traffic information into consideration."

[0244] This system allows users to efficiently obtain information by voice without operating the vehicle, improving safety while driving.

[0245] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0246] Step 1:

[0247] Capture voice input

[0248] Input: User utterance

[0249] Output: Audio data

[0250] Specific operation: When a user speaks to the voice interface device, the terminal (interface device in the autonomous mobile body) captures the voice using a microphone. The captured voice is sent as digital data to the server via the communication module.

[0251] Step 2:

[0252] Converting audio data to text

[0253] Input: Audio data

[0254] Output: Text data

[0255] Specific operation: The server inputs the received voice data into the voice recognition API. The voice recognition API performs spectral analysis of the voice signal and generates corresponding text data. For example, if a user says, "Tell me the shortest route to my house," the voice data is converted into text data saying, "Tell me the shortest route to my house."

[0256] Step 3:

[0257] Natural Language Processing and Response Generation

[0258] Input: Text data

[0259] Output: Response text

[0260] Specific operation: The server inputs the converted text data into a natural language processing engine. The natural language processing engine analyzes the text data and understands the user's instructions and questions. For example, in response to an input such as "What is the shortest route to my house?", the engine calls a map information processing engine and generates appropriate route guidance. The generated response text is "The shortest route from my current location to my house is..."

[0261] Step 4:

[0262] Generate response audio data

[0263] Input: Response text

[0264] Output: Audio data

[0265] Specific operation: The server passes the generated response text to the speech synthesis API. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech data. For example, a text response such as "The shortest route from your current location to your home is..." is converted into speech data.

[0266] Step 5:

[0267] Sending and playing audio data

[0268] Input: Audio data

[0269] Output: Audio output to the user

[0270] Specific operation: The generated voice data is sent from the server to the voice interface device via the communication module of the autonomous mobile device. The terminal plays the received voice data through a speaker and provides a response to the user. For example, the user can hear a voice response such as, "The shortest route from your current location to your home is..."

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[0273] Overall system configuration

[0274] This system consists of a user device (such as Bluetooth earphones or audio glasses), a server located in the cloud, and an emotion engine. The user asks questions or gives instructions via voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[0275] Detailed program processing

[0276] Server Processing

[0277] 1. Receiving voice input

[0278] The server receives the voice data sent from the device, which is first decoded and converted into an analyzable format.

[0279] 2. Converting voice data to text

[0280] The server converts the voice data into text data using a speech recognition API, performs spectral analysis of the voice signal, and extracts the spoken content as a string of characters.

[0281] 3. Emotion Analysis

[0282] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[0283] 4. Natural Language Processing and Response Generation

[0284] The server inputs the text data into a natural language processing engine to analyze the user's intent. It also takes into account the results of sentiment analysis and adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[0285] 5. Generating Audio Data

[0286] The server then passes the generated text response to a speech synthesis API, which converts it into audio data, adjusting intonation and tone based on the results of sentiment analysis.

[0287] 6. Sending audio data

[0288] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[0289] Terminal handling

[0290] 1. Capture voice input

[0291] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[0292] 2. Receiving and playing audio data

[0293] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[0294] Specific examples

[0295] Example 1: Checking the schedule

[0296] 1. User: "What's my schedule for tomorrow?"

[0297] 2. Device: Uses microphone to capture audio and send it to the server.

[0298] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[0299] 4. Server: The emotion engine recognizes that the user is nervous.

[0300] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10:00 AM." This is converted into voice data using a speech synthesis API in a gentle tone.

[0301] 6. Server: Sends audio data to the device.

[0302] 7. Terminal: Play audio data.

[0303] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0304] Example 2: Checking the weather forecast

[0305] 1. User: "What's the weather like today?"

[0306] 2. Device: Uses microphone to capture audio and send it to the server.

[0307] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[0308] 4. Server: The emotion engine recognizes that the user is calm.

[0309] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[0310] 6. Server: Sends audio data to the device.

[0311] 7. Terminal: Play audio data.

[0312] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0313] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[0314] The processing flow will be explained below.

[0315] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[0316] Detailed program processing

[0317] Step 1:

[0318] The user speaks into the Bluetooth earphones or audio glasses to ask questions or give instructions, such as "Tell me the weather today."

[0319] Step 2:

[0320] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[0321] Step 3:

[0322] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[0323] Step 4:

[0324] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[0325] Step 5:

[0326] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[0327] Step 6:

[0328] The server inputs the text data into a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. Taking into account the results of sentiment analysis, the server adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[0329] Step 7:

[0330] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, it generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0331] Step 8:

[0332] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API generates natural-sounding speech by taking into account the grammatical structure and intonation of the text. It also adjusts the intonation and tone based on the results of sentiment analysis.

[0333] Step 9:

[0334] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[0335] Step 10:

[0336] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[0337] Step 11:

[0338] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0339] Specific examples

[0340] Example 1: Checking the schedule

[0341] 1. User: "What's my schedule for tomorrow?"

[0342] 2. Device: Uses microphone to capture audio and send it to the server.

[0343] 3. Server: Use a speech recognition API to convert the speech into text, saying "What's my schedule for tomorrow?"

[0344] 4. Server: The emotion engine recognizes that the user is nervous.

[0345] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10 AM." The response is converted into voice data using a speech synthesis API in a gentle tone.

[0346] 6. Server: Sends the audio data to the device.

[0347] 7. Terminal: Plays audio data.

[0348] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0349] Example 2: Checking the weather forecast

[0350] 1. User: "What's the weather like today?"

[0351] 2. Device: Uses microphone to capture audio and send it to the server.

[0352] 3. Server: Use the speech recognition API to convert the speech into text, saying "What's the weather like today?"

[0353] 4. Server: The emotion engine recognizes that the user is calm.

[0354] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[0355] 6. Server: Sends the audio data to the device.

[0356] 7. Terminal: Plays audio data.

[0357] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0358] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[0359] Example 2

[0360] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] While modern voice assistant systems typically provide responses based on user utterances, they rarely consider the user's emotional state. As a result, appropriate responses are not provided based on the user's emotional state, resulting in a poor user experience. In addition, the system's inability to recognize the user's emotions makes it difficult to provide more human-like interactions.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0363] In this invention, the server includes a means for converting voice input into text data, a means for recognizing the user's emotions based on the text data, and a means for analyzing the user's intentions and generating an appropriate response according to the emotions, thereby enabling a more natural and human-like response that takes the user's emotions into consideration.

[0364] A "user" is a person who uses the system to provide voice input and receive responses.

[0365] "Voice input" refers to a voice signal of a question or instruction verbally given by the user.

[0366] A "capturing means" is a device or method for converting a user's voice input into digital data and acquiring it.

[0367] A "server" is a central object that works in conjunction with terminals via a network to analyze voice data, convert it into text, analyze emotions, and generate responses.

[0368] "Text data" is voice data converted into a character string, and is generated by voice recognition.

[0369] A "converting means" is a technique or device used to convert voice data into text data or text data into voice data.

[0370] "Means for recognizing emotions" refers to technologies and algorithms that analyze text data and voice data to determine a user's emotional state.

[0371] "Means for analyzing intent" refers to techniques and methods for understanding what a user wants based on the content of their speech and their emotional state, and generating an appropriate response.

[0372] "Means for generating a response" refers to techniques and methods for generating appropriate information or answers in response to the analyzed user's intentions and emotions.

[0373] "Means for reproducing audio data" refers to a device or method for providing the generated audio data to the user in a reproducible format.

[0374] The present invention aims to develop a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connections, and receive responses by voice. By further combining this with an emotion engine, the system can recognize the user's emotions and generate more appropriate responses.

[0375] This system consists of a user's device and a server and emotion engine located on the cloud. The user speaks questions or commands through the device, and the voice data is sent to the server via the network. The server converts the voice data into text data, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays back the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[0376] Hardware and software used

[0377] 1. Terminal

[0378] Bluetooth earphones and audio glasses are used to capture the user's voice. These devices use a built-in microphone to convert audio input into digital data, which is then sent to a server over the internet.

[0379] 2. Server

[0380] Speech recognition API: Used to convert voice data into text data. Examples include Google Cloud Speech-to-Text and Azure® Speech Service.

[0381] Natural language processing engines: Used to analyze text data and identify user intent. Examples include OpenAI GPT-3 and Amazon Lex.

[0382] Emotion engines: Used to analyze the user's emotional state. Examples include IBM Watson® Tone Analyzer and Affectiva.

[0383] Text-to-speech APIs: Used to convert text data into speech data, such as Amazon Polly and Google Cloud Text-to-Speech.

[0384] Specific examples

[0385] Example 1: Checking the schedule

[0386] 1. User: "What's my schedule for tomorrow?"

[0387] 2. Device: Uses microphone to capture audio and send it to the server.

[0388] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's tomorrow's schedule?"

[0389] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is nervous.

[0390] 5. Server: Generate a response using a natural language processing engine (OpenAI GPT-3) saying, "We have a meeting tomorrow at 10:00 AM." Convert this into voice data using a speech synthesis API (Amazon Polly) in a gentle tone.

[0391] 6. Server: Sends audio data to the device.

[0392] 7. Terminal: Play audio data.

[0393] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0394] Example 2: Checking the weather forecast

[0395] 1. User: "What's the weather like today?"

[0396] 2. Device: Uses microphone to capture audio and send it to the server.

[0397] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's the weather like today?"

[0398] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is calm.

[0399] 5. Server: A natural language processing engine (OpenAI GPT-3) generates a response, stating, "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API (Amazon Polly) in a normal tone.

[0400] 6. Server: Sends audio data to the device.

[0401] 7. Terminal: Play audio data.

[0402] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0403] This system allows users to obtain information efficiently and naturally without manually operating the device, and to receive responses that correspond to their emotional state at the time. This provides great convenience as it does not require users to interrupt other tasks and can respond in a way that takes into account the user's emotions.

[0404] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0405] Step 1:

[0406] Capturing voice input

[0407] The user puts on Bluetooth earphones or audio glasses and asks questions or gives instructions such as "What's the weather like today?"

[0408] Input: User speech.

[0409] The device uses a built-in microphone to capture the user's speech as audio data, which is then converted into a digital format.

[0410] Output: Audio data in digital format.

[0411] Step 2:

[0412] Sending audio data

[0413] Input: Audio data in digital format.

[0414] The device transmits the captured audio data to a server over a secure network connection.

[0415] Output: The audio data sent to the server.

[0416] Step 3:

[0417] Receiving and decoding audio data

[0418] Input: Audio data sent from the device.

[0419] The server receives the audio data sent from the device. The received audio data is decoded into an analyzable format. This decoding process makes the audio data easier to handle as a digital signal.

[0420] Output: Decoded audio data in a parsable format.

[0421] Step 4:

[0422] Converting audio data to text

[0423] Input: Decoded audio data.

[0424] The server calls a speech recognition API (such as Google Cloud Speech-to-Text) to convert the audio data into text data. During this process, the audio signal is spectrally analyzed and the spoken content is extracted as a string of characters.

[0425] Output: Text data (e.g. "What's the weather like today?").

[0426] Step 5:

[0427] Emotion Analysis

[0428] Input: Text and audio data.

[0429] The server uses an emotion engine (such as IBM Watson Tone Analyzer) to analyze the tone, rate, and intonation of the voice data, thereby recognizing the user's emotional state, such as whether they are nervous, excited, or calm.

[0430] Output: The user's emotional state (e.g., calm).

[0431] Step 6:

[0432] Natural Language Processing and Response Generation

[0433] Input: Text data and sentiment analysis results.

[0434] The server inputs the text data into a natural language processing engine (such as OpenAI GPT-3) to analyze the user's intent. Taking into account the results of sentiment analysis, the server generates an appropriate response and tone. For example, if the user asks, "What's the weather like today?", the response will be, "It's cloudy with occasional rain today, with a maximum temperature of 22 degrees," and if the user is calm, the response should be in a normal tone.

[0435] Output: Response text (e.g., "Today it will be cloudy with occasional rain, with a high of 22 degrees.").

[0436] Step 7:

[0437] Generating audio data

[0438] Input: Response text and sentiment analysis results.

[0439] The server then passes the generated text response to a speech synthesis API (such as Amazon Polly) and converts it into speech data. During speech synthesis, the server also uses the results of emotion analysis to set appropriate intonation and tone.

[0440] Output: Response audio data.

[0441] Step 8:

[0442] Encoding and transmitting audio data

[0443] Input: Response audio data.

[0444] The server encodes the generated voice data for transmission to the terminal, and the encoded voice data is sent back to the terminal via the network.

[0445] Output: The audio data sent to the device.

[0446] Step 9:

[0447] Receiving and playing audio data

[0448] Input: Audio data sent from the server.

[0449] The device receives the audio data sent from the server, decodes it into a playable format, and plays the decoded audio through the device's speaker.

[0450] Output: The audio data to be played (e.g., "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees.").

[0451] (Application example 2)

[0452] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0453] Modern voice interfaces are capable of generating appropriate responses to user inputs. However, these systems are unable to consider the user's emotional state, and the responses are often inappropriate for the user's situation. This results in a poor user experience, as the system is unable to provide intuitively desirable responses based on the user's emotional state. Furthermore, it is difficult to provide personalized suggestions based on the user's emotional state, especially in services such as food delivery.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0455] In this invention, the server includes means for capturing voice input, means for converting the voice input from voice to text data, means for recognizing the emotional state of the user, means for generating an appropriate response from the text data taking the emotional state into consideration, means for converting the generated response from text to voice data, and means for transmitting the voice data to the user, thereby enabling personalized content suggestions based on the user's emotional state.

[0456] "Means for capturing audio input" refers to a microphone or sensor that detects the audio emitted by the user and converts it into digital data.

[0457] The "means for transmitting audio input to a server" is a component that has the function of transmitting captured audio data to a remote server via a communication network such as the Internet.

[0458] "Means for converting voice input into text data" refers to software or hardware that uses voice recognition technology to convert received voice data into text.

[0459] "Means for generating an appropriate response" refers to an algorithm or program that analyzes the user's input intent and generates answers or suggestions to the user's questions or requests based on the analysis results.

[0460] The "means for converting a response into voice data" is a system for converting the generated text-format response into voice data using a natural language processing engine or voice synthesis technology.

[0461] The "means for transmitting voice data to a user" refers to a component having a function for transmitting voice data to a user's terminal via a network.

[0462] "Means for reproducing audio data" refers to a speaker or earphone for reproducing audio data received at a user terminal.

[0463] The "means for recognizing emotional state" is a system that includes machine learning models and algorithms for analyzing tone, intonation, facial expressions, etc. from a user's voice data to identify the user's emotional state.

[0464] "Means for generating responses based on emotional state" refers to a program or algorithm for generating responses that take into account the user's perceived emotional state and have a tone and content appropriate to the user's situation.

[0465] A "means for suggesting personalized content based on a user's emotional state" is a system or algorithm for providing appropriate, individualized content or suggestions based on a user's emotional state.

[0466] The present invention is a voice dialogue system including user emotion recognition, particularly adapted for food delivery applications.

[0467] Overall system configuration

[0468] The system mainly consists of the following components:

[0469] 1. A device that captures the user's voice

[0470] 2. Means of transmitting audio data from the device to the server

[0471] 3. Server-side speech recognition, natural language processing, and emotion recognition

[0472] 4. Emotion-based response generation

[0473] 5. A means of converting the response into audio data and sending it to the device

[0474] 6. How to play audio data on the device

[0475] Program processing explanation

[0476] User side (terminal)

[0477] A user inputs voice using a device such as a smartphone or smart glasses. The device captures the voice using a built-in microphone and transmits the voice data to a server via a network.

[0478] Server side

[0479] The server does the following:

[0480] 1. Speech recognition: Use a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert voice data into text data.

[0481] 2. Emotion Recognition: An emotion engine (e.g., cognitive emotion recognition model) analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from text and voice data.

[0482] 3. Natural Language Processing: A natural language processing engine (e.g., OpenAI GPT model) is used to understand the user's intent and generate an appropriate response based on that. The generated response is adjusted in content and tone based on the user's emotional state.

[0483] 4. Convert the response to speech: Use a text-to-speech API (e.g., Amazon Polly) to convert the text response to audio data.

[0484] 5. Sending voice data: The generated voice data is sent again to the user's terminal via the network.

[0485] Actual system usage example

[0486] For example, the system works in the following example:

[0487] What to do if you're tired

[0488] 1. User: "I feel kind of tired..."

[0489] 2. Device: Uses microphone to capture audio and send it to the server.

[0490] 3. Server: Converts the speech into text using a speech recognition API. The server recognizes the phrase "I feel tired."

[0491] 4. Server: The emotion engine recognizes that the user is tired.

[0492] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as, "Cheer up, how about a nutritious salad bowl?"

[0493] 6. Server: Convert into voice data using speech synthesis API.

[0494] 7. Server: Sends audio data to the device.

[0495] 8. Device: Play audio data. The audio says, "Cheer up! How about a nutritious salad bowl?"

[0496] What to do if things are calm

[0497] 1. User: "What should I have for lunch?"

[0498] 2. Device: Uses microphone to capture audio and send it to the server.

[0499] 3. Server: Converts speech to text using a speech recognition API. Recognizes "What should I have for lunch?"

[0500] 4. Server: The emotion engine recognizes that the user is calm.

[0501] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as "You look great today! How about some salmon sushi?"

[0502] 6. Server: Convert into voice data using speech synthesis API.

[0503] 7. Server: Sends audio data to the device.

[0504] 8. Device: Play the audio data. The audio will say, "You look great today! How about some salmon sushi?"

[0505] Prompt Sentence Examples

[0506] Example prompt 1: "Generate a dialogue that suggests a menu to help the user relax when they are tired."

[0507] Example prompt 2: "Generate a dialogue that suggests an energizing meal for the user when they are calm."

[0508] In this way, a system can be implemented that provides personalized responses that take into account the user's emotions.

[0509] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0510] Step 1:

[0511] The user performs voice input. The user issues instructions or asks questions by voice to a device such as a smartphone or smart glasses. The input data is the user's voice (voice signal).

[0512] Step 2:

[0513] The device captures audio input by using the device's built-in microphone to convert the user's voice into digital data, which is raw audio data (such as WAV or MP3 format).

[0514] Step 3:

[0515] The device sends the captured audio data to the server. The digital audio data is sent to the server via a communication network (Wi-Fi or mobile data communication). The input is digital audio data, and the output is the same data transferred to the server.

[0516] Step 4:

[0517] The server converts the received voice data into text using a speech recognition API, such as Google Cloud Speech-to-Text, which analyzes the voice signal and generates a corresponding string of characters. The input is digital voice data, and the output is text instructions or questions.

[0518] Step 5:

[0519] The server inputs the text data into the emotion engine to recognize the user's emotional state. The emotion engine uses a machine learning model to analyze the user's emotions (happiness, sadness, anger, etc.) from the text. The input is text data, and the output is metadata that indicates the user's emotional state.

[0520] Step 6:

[0521] The server uses a natural language processing engine to analyze the user's intent and generate an appropriate response based on that. It also takes into account the results of sentiment analysis to adjust the content and tone of the response. For example, it uses the OpenAI GPT model to generate answers and suggestions to the user's questions. The input is text data and sentiment metadata, and the output is a sentiment-adjusted text response.

[0522] Step 7:

[0523] The server converts the generated text response into audio data using a speech synthesis API. Services such as Amazon Polly are used to synthesize the text into natural-sounding speech. The input is the text response, and the output is audio data (in a playable format).

[0524] Step 8:

[0525] The server encodes the voice data and sends it to the terminal. The generated voice data is then sent back to the user's terminal via the network. The input is the encoded voice data, and the output is the same data transferred to the terminal.

[0526] Step 9:

[0527] The device plays the received audio data. The audio data is played to the user using the device's speaker or earphones. The input is digital audio data, and the output is played back in a form that the user can hear as sound.

[0528] These steps allow users to operate a food delivery application using only their voice and receive personalized suggestions based on their emotions.

[0529] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0531] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0532] [Second embodiment]

[0533] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0534] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0535] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0536] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0537] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0539] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0540] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0541] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0542] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0543] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0544] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0545] The present invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. Specific embodiments of this system are described below.

[0546] Overall system configuration

[0547] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[0548] Detailed program processing

[0549] Server Processing

[0550] 1. Receiving voice input

[0551] The server receives the voice data sent by the user and passes it to the voice recognition API.

[0552] 2. Converting voice data to text

[0553] The server converts the voice data into text data using a voice recognition API, which performs, for example, spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[0554] 3. Natural Language Processing and Response Generation

[0555] The server then inputs the resulting text data into a natural language processing engine, which analyzes the voice commands and questions and generates an appropriate response. For example, if a user asks for the weather forecast, the engine queries a weather forecast API and generates a response based on the results.

[0556] 4. Generating Audio Data

[0557] The server then passes the generated text response to a speech synthesis API, which converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[0558] 5. Sending audio data

[0559] The server transmits the generated voice data to the user's terminal via the network.

[0560] Terminal handling

[0561] 1. Capture voice input

[0562] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[0563] 2. Receiving and playing audio data

[0564] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[0565] Specific examples

[0566] Example 1: Checking the schedule

[0567] 1. User: "What's my schedule for tomorrow?"

[0568] 2. Device: Uses microphone to capture audio and send it to the server.

[0569] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[0570] 4. Server: The natural language processing engine calls the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[0571] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0572] 6. Server: Sends audio data to the device.

[0573] 7. Terminal: Play audio data.

[0574] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0575] Example 2: Checking the weather forecast

[0576] 1. User: "What's the weather like today?"

[0577] 2. Device: Uses microphone to capture audio and send it to the server.

[0578] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[0579] 4. Server: The natural language processing engine calls the weather forecast API and generates a response: "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[0580] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0581] 6. Server: Sends audio data to the device.

[0582] 7. Terminal: Play audio data.

[0583] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0584] This system allows users to efficiently obtain information by voice without manual input or visual confirmation, providing great convenience especially when hands are busy or during tasks that require visual concentration.

[0585] The processing flow will be explained below.

[0586] Step 1:

[0587] The user inputs voice into the Bluetooth earphones or audio glasses. The user's speech is a question or instruction such as "Tell me today's weather."

[0588] Step 2:

[0589] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[0590] Step 3:

[0591] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[0592] Step 4:

[0593] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[0594] Step 5:

[0595] The server passes the text data to a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. For example, it may determine that the request is for a weather forecast. The natural language processing engine analyzes the input text data and determines an appropriate response.

[0596] Step 6:

[0597] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, the server generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0598] Step 7:

[0599] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[0600] Step 8:

[0601] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the encoded voice data to the terminal via the network.

[0602] Step 9:

[0603] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[0604] Step 10:

[0605] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0606] This process flow allows users to obtain information and execute instructions using only their voice, without having to manually operate the device. This allows users to solve problems efficiently without interrupting other tasks.

[0607] Example 1

[0608] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0609] Conventional voice interface systems often require users to manually input information or look at a screen when obtaining it through voice, limiting their convenience. Furthermore, these systems are only effective in specific locations and situations, making flexible use difficult. Furthermore, if the accuracy of speech recognition and response generation is low, users may not receive satisfactory service.

[0610] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0611] In this invention, the server includes means for acquiring voice input from a user, means for transmitting the voice input to a processing device, means for converting the voice input into text data, means for generating an appropriate response based on the text data, means for converting the response into voice data, means for transmitting the voice data to the user, and means for playing back the voice data. This allows the user to efficiently acquire information in a natural, interactive format without having to manually input or check a screen. Specifically, by using a voice recognition API and a natural language processing engine, the server can analyze the user's intent with high accuracy and generate and provide an appropriate response.

[0612] A "means for obtaining voice input" is a device that senses a user's speech and converts it into a voice signal.

[0613] The "means for transmitting to a processing unit" is a device or function that transmits the captured audio signal as digital data to a central processing unit via a network.

[0614] "Means for converting into text data" refers to a function or device that analyzes a voice signal and converts it into text data.

[0615] The "means for generating an appropriate response" refers to a function or device that analyzes the user's intention based on character data and generates a response in natural language.

[0616] The "means for converting into voice data" refers to a function or device that converts the generated text response into voice data.

[0617] The "means for transmitting to the user" refers to a function or device that transmits voice data to the user's terminal via a network.

[0618] The "means for reproducing audio data" is a device such as a speaker that reproduces audio data received by the user's terminal.

[0619] The "means for analyzing in natural language" refers to a function or device for analyzing acquired text data and understanding the intent of the user's questions and instructions.

[0620] An "external language processing engine" is an engine with natural language processing capabilities that is provided through a cloud service or external API.

[0621] This invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connection, and receive responses by voice. Here we will show how to specifically implement this system.

[0622] Overall system configuration

[0623] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[0624] Hardware and software used

[0625] Devices: Bluetooth earphones, audio glasses with built-in mobile connection

[0626] Server: High-performance cloud computing services

[0627] Speech recognition API: Google Speech-to-Text, etc.

[0628] Natural language processing engine: OpenAI GPT-3, etc.

[0629] Speech synthesis API: Google Text-to-Speech, etc.

[0630] Network: Internet connection (Wi-Fi, mobile data)

[0631] Processing of various data

[0632] Receiving and processing voice input

[0633] User: Ask questions and give instructions using voice, just like in everyday conversation.

[0634] The device captures the user's voice using a built-in microphone, and the voice data is compressed through digital signal processing.

[0635] The terminal transmits the compressed audio data to the server via the network.

[0636] Speech-to-text transcription and analysis

[0637] The server receives the audio data over the network.

[0638] The server converts the received voice data into text using the Google Speech-to-Text API.

[0639] The server passes the converted text data to a natural language processing engine such as OpenAI GPT-3, which analyzes the user's intent and generates an appropriate response.

[0640] Response generation and speech synthesis

[0641] The server passes the generated text response to a speech synthesis API, such as the Google Text-to-Speech API, and converts it into audio data.

[0642] The server encodes the generated voice data and transmits it to the user's terminal.

[0643] Playing audio

[0644] The terminal decodes the received audio data and plays it on the speaker.

[0645] The user hears the response from the server through the played audio.

[0646] Specific examples

[0647] Example 1: Checking the schedule

[0648] 1. User: "What's my schedule for tomorrow?"

[0649] 2. Device: Uses microphone to capture audio and send it to the server.

[0650] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[0651] 4. Server: The natural language processing engine queries the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[0652] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0653] 6. Server: Sends audio data to the device.

[0654] 7. Terminal: Play audio data.

[0655] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0656] Example 2: Checking the weather forecast

[0657] 1. User: "What's the weather like today?"

[0658] 2. Device: Uses microphone to capture audio and send it to the server.

[0659] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[0660] 4. Server: The natural language processing engine queries the weather API and generates a response: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[0661] 5. Server: Convert the text response into audio data using a speech synthesis API.

[0662] 6. Server: Sends audio data to the device.

[0663] 7. Terminal: Play audio data.

[0664] 8. User: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[0665] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0666] Step 1:

[0667] The user asks questions or gives instructions by voice.

[0668] Input: User utterance "What's the weather like today?"

[0669] Specific actions: The user speaks through Bluetooth earphones or audio glasses.

[0670] Step 2:

[0671] The terminal captures the user's speech.

[0672] Input: User's spoken utterance

[0673] What it does: It uses a built-in microphone to capture the user's speech as digital audio data.

[0674] Output: Digital audio data

[0675] Step 3:

[0676] The terminal transmits the captured audio data to the server.

[0677] Input: Digital audio data

[0678] What it does: It compresses digital audio data and sends it over the network to a server.

[0679] Output: Compressed audio data

[0680] Step 4:

[0681] The server passes the received voice data to the voice recognition API.

[0682] Input: Compressed audio data

[0683] Specific operation: Decodes the audio data and passes it to a speech recognition API such as the Google Speech-to-Text API.

[0684] Output: Text data "What's the weather like today?"

[0685] Step 5:

[0686] The server inputs the data converted into text by the speech recognition API into a natural language processing engine.

[0687] Input: Text data "What's the weather like today?"

[0688] What it does: Provides text data as prompts to a natural language processing engine such as OpenAI GPT-3.

[0689] Output: Response "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0690] Step 6:

[0691] The server passes the generated response text to the speech synthesis API.

[0692] Input: Response "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[0693] What it does: Passes the response text to a speech synthesis API, such as the Google Text-to-Speech API, to generate natural-sounding speech.

[0694] Output: Audio data

[0695] Step 7:

[0696] The server transmits the generated voice data to the user's terminal.

[0697] Input: Audio data

[0698] Specific operations: Encodes audio data and sends it over the network to the user's device.

[0699] Output: Audio data

[0700] Step 8:

[0701] The terminal decodes the received audio data and plays it on the speaker.

[0702] Input: Received audio data

[0703] Specific behavior: Decodes audio data and plays it to the user through the built-in speaker.

[0704] Output: Audio played: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[0705] (Application example 1)

[0706] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0707] Current voice interfaces for autonomous vehicles have difficulty accurately analyzing user voice input and providing appropriate information. Furthermore, there is a lack of technology that allows drivers to efficiently operate autonomous vehicles through voice commands, making it difficult to balance safe operation with information provision. Therefore, there is a demand for a system that can improve the accuracy of voice interfaces within autonomous vehicles and provide appropriate information needed during operation.

[0708] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0709] In this invention, the server includes means for capturing voice input from a user, means for transmitting the voice input to the server, and means for converting the voice input into text data, thereby enabling generation of appropriate route guidance based on position information within the autonomous mobile body.

[0710] "Voice input" is voice data uttered by the user through a microphone.

[0711] A "server" is a computer system that analyzes voice and text data over a network and generates an appropriate response.

[0712] "Text data" is character information converted by speech recognition.

[0713] A "response" is information or instructions generated based on voice input.

[0714] "Audio data" is data in an audio format that is converted from text data by voice synthesis.

[0715] An "autonomous mobile object" is a machine such as a vehicle or robot that moves autonomously without human operation.

[0716] "Route guidance" refers to instructions that include route information to a destination.

[0717] "Means for analyzing in natural language" refers to methods and technologies for analyzing and understanding voice input using language processing technology.

[0718] A "map information processing engine" is a system that analyzes map data and provides route guidance and identifies the current location.

[0719] Overall system configuration

[0720] This system consists of a voice interface device installed in the autonomous vehicle and an external cloud server. Users can communicate with the autonomous vehicle through the voice interface and receive various information and instructions.

[0721] Hardware Configuration

[0722] Voice interface device for autonomous mobile body

[0723] The audio interface device includes the following hardware:

[0724] Microphone: Captures audio input.

[0725] Speaker: Plays back the generated audio data.

[0726] Communication module: Sends and receives data to and from the server.

[0727] server

[0728] It exists on the cloud and has the following functions:

[0729] Speech Recognition API: Converts voice data into text.

[0730] Natural language processing engine: Analyzes text and generates responses.

[0731] Map information processing engine: Generates route guidance based on location information.

[0732] Speech synthesis API: Converts text into audio data.

[0733] Software configuration and data processing

[0734] 1. Capture voice input

[0735] When a user speaks into the voice interface device, a built-in microphone captures the voice and transmits the data to a server.

[0736] 2. Converting voice data to text

[0737] The server converts the transmitted voice data into text data using a speech recognition API, which performs spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[0738] 3. Natural Language Processing and Response Generation

[0739] The text data is fed into a natural language processing engine, which analyzes the question or instruction and generates an appropriate response. For example, if the user requests route guidance, a map information processing engine is called to calculate the optimal route.

[0740] 4. Generate response audio data

[0741] The generated response text is passed to a speech synthesis API and converted into audio data, which takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[0742] 5. Sending and Playing Audio Data

[0743] The generated voice data is transmitted to the voice interface device through the communication module of the autonomous mobile body and played back through a speaker.

[0744] Specific examples

[0745] 1. Providing route guidance

[0746] User: "What's the quickest route to my house?"

[0747] Server: Converts speech to text using speech recognition API.

[0748] Server: The natural language processing engine analyzes the request "Tell me the shortest route to my house" and calls the map information processing engine to obtain route information.

[0749] Server: Convert the generated route guidance into voice data using a speech synthesis API.

[0750] Autonomous mobile unit: Receives and plays back audio data.

[0751] 2. Examples of prompts

[0752] "Please tell me the shortest route to my destination. Please specify the fastest route. Starting point: current location, destination: home. Please also take current traffic information into consideration."

[0753] This system allows users to efficiently obtain information by voice without operating the vehicle, improving safety while driving.

[0754] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0755] Step 1:

[0756] Capture voice input

[0757] Input: User utterance

[0758] Output: Audio data

[0759] Specific operation: When a user speaks to the voice interface device, the terminal (interface device in the autonomous mobile body) captures the voice using a microphone. The captured voice is sent as digital data to the server via the communication module.

[0760] Step 2:

[0761] Converting audio data to text

[0762] Input: Audio data

[0763] Output: Text data

[0764] Specific operation: The server inputs the received voice data into the voice recognition API. The voice recognition API performs spectral analysis of the voice signal and generates corresponding text data. For example, if a user says, "Tell me the shortest route to my house," the voice data is converted into text data saying, "Tell me the shortest route to my house."

[0765] Step 3:

[0766] Natural Language Processing and Response Generation

[0767] Input: Text data

[0768] Output: Response text

[0769] Specific operation: The server inputs the converted text data into a natural language processing engine. The natural language processing engine analyzes the text data and understands the user's instructions and questions. For example, in response to an input such as "What is the shortest route to my house?", the engine calls a map information processing engine and generates appropriate route guidance. The generated response text is "The shortest route from my current location to my house is..."

[0770] Step 4:

[0771] Generate response audio data

[0772] Input: Response text

[0773] Output: Audio data

[0774] Specific operation: The server passes the generated response text to the speech synthesis API. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech data. For example, a text response such as "The shortest route from your current location to your home is..." is converted into speech data.

[0775] Step 5:

[0776] Sending and playing audio data

[0777] Input: Audio data

[0778] Output: Audio output to the user

[0779] Specific operation: The generated voice data is sent from the server to the voice interface device via the communication module of the autonomous mobile device. The terminal plays the received voice data through a speaker and provides a response to the user. For example, the user can hear a voice response such as, "The shortest route from your current location to your home is..."

[0780] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0781] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[0782] Overall system configuration

[0783] This system consists of a user device (such as Bluetooth earphones or audio glasses), a server located in the cloud, and an emotion engine. The user asks questions or gives instructions via voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[0784] Detailed program processing

[0785] Server Processing

[0786] 1. Receiving voice input

[0787] The server receives the voice data sent from the device, which is first decoded and converted into an analyzable format.

[0788] 2. Converting voice data to text

[0789] The server converts the voice data into text data using a speech recognition API, performs spectral analysis of the voice signal, and extracts the spoken content as a string of characters.

[0790] 3. Emotion Analysis

[0791] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[0792] 4. Natural Language Processing and Response Generation

[0793] The server inputs the text data into a natural language processing engine to analyze the user's intent. It also takes into account the results of sentiment analysis and adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[0794] 5. Generating Audio Data

[0795] The server then passes the generated text response to a speech synthesis API, which converts it into audio data, adjusting intonation and tone based on the results of sentiment analysis.

[0796] 6. Sending audio data

[0797] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[0798] Terminal handling

[0799] 1. Capture voice input

[0800] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[0801] 2. Receiving and playing audio data

[0802] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[0803] Specific examples

[0804] Example 1: Checking the schedule

[0805] 1. User: "What's my schedule for tomorrow?"

[0806] 2. Device: Uses microphone to capture audio and send it to the server.

[0807] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[0808] 4. Server: The emotion engine recognizes that the user is nervous.

[0809] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10:00 AM." This is converted into voice data using a speech synthesis API in a gentle tone.

[0810] 6. Server: Sends audio data to the device.

[0811] 7. Terminal: Play audio data.

[0812] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0813] Example 2: Checking the weather forecast

[0814] 1. User: "What's the weather like today?"

[0815] 2. Device: Uses microphone to capture audio and send it to the server.

[0816] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[0817] 4. Server: The emotion engine recognizes that the user is calm.

[0818] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[0819] 6. Server: Sends audio data to the device.

[0820] 7. Terminal: Play audio data.

[0821] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0822] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[0823] The processing flow will be explained below.

[0824] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[0825] Detailed program processing

[0826] Step 1:

[0827] The user speaks into the Bluetooth earphones or audio glasses to ask questions or give instructions, such as "Tell me the weather today."

[0828] Step 2:

[0829] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[0830] Step 3:

[0831] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[0832] Step 4:

[0833] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[0834] Step 5:

[0835] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[0836] Step 6:

[0837] The server inputs the text data into a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. Taking into account the results of sentiment analysis, the server adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[0838] Step 7:

[0839] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, it generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0840] Step 8:

[0841] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API generates natural-sounding speech by taking into account the grammatical structure and intonation of the text. It also adjusts the intonation and tone based on the results of sentiment analysis.

[0842] Step 9:

[0843] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[0844] Step 10:

[0845] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[0846] Step 11:

[0847] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[0848] Specific examples

[0849] Example 1: Checking the schedule

[0850] 1. User: "What's my schedule for tomorrow?"

[0851] 2. Device: Uses microphone to capture audio and send it to the server.

[0852] 3. Server: Use a speech recognition API to convert the speech into text, saying "What's my schedule for tomorrow?"

[0853] 4. Server: The emotion engine recognizes that the user is nervous.

[0854] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10 AM." The response is converted into voice data using a speech synthesis API in a gentle tone.

[0855] 6. Server: Sends the audio data to the device.

[0856] 7. Terminal: Plays audio data.

[0857] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0858] Example 2: Checking the weather forecast

[0859] 1. User: "What's the weather like today?"

[0860] 2. Device: Uses microphone to capture audio and send it to the server.

[0861] 3. Server: Use the speech recognition API to convert the speech into text, saying "What's the weather like today?"

[0862] 4. Server: The emotion engine recognizes that the user is calm.

[0863] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[0864] 6. Server: Sends the audio data to the device.

[0865] 7. Terminal: Plays audio data.

[0866] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0867] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[0868] Example 2

[0869] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0870] While modern voice assistant systems typically provide responses based on user utterances, they rarely consider the user's emotional state. As a result, appropriate responses are not provided based on the user's emotional state, resulting in a poor user experience. In addition, the system's inability to recognize the user's emotions makes it difficult to provide more human-like interactions.

[0871] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0872] In this invention, the server includes a means for converting voice input into text data, a means for recognizing the user's emotions based on the text data, and a means for analyzing the user's intentions and generating an appropriate response according to the emotions, thereby enabling a more natural and human-like response that takes the user's emotions into consideration.

[0873] A "user" is a person who uses the system to provide voice input and receive responses.

[0874] "Voice input" refers to a voice signal of a question or instruction verbally given by the user.

[0875] A "capturing means" is a device or method for converting a user's voice input into digital data and acquiring it.

[0876] A "server" is a central object that works in conjunction with terminals via a network to analyze voice data, convert it into text, analyze emotions, and generate responses.

[0877] "Text data" is voice data converted into a character string, and is generated by voice recognition.

[0878] A "converting means" is a technique or device used to convert voice data into text data or text data into voice data.

[0879] "Means for recognizing emotions" refers to technologies and algorithms that analyze text data and voice data to determine a user's emotional state.

[0880] "Means for analyzing intent" refers to techniques and methods for understanding what a user wants based on the content of their speech and their emotional state, and generating an appropriate response.

[0881] "Means for generating a response" refers to techniques and methods for generating appropriate information or answers in response to the analyzed user's intentions and emotions.

[0882] "Means for reproducing audio data" refers to a device or method for providing the generated audio data to the user in a reproducible format.

[0883] The present invention aims to develop a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connections, and receive responses by voice. By further combining this with an emotion engine, the system can recognize the user's emotions and generate more appropriate responses.

[0884] This system consists of a user's device and a server and emotion engine located on the cloud. The user speaks questions or commands through the device, and the voice data is sent to the server via the network. The server converts the voice data into text data, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays back the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[0885] Hardware and software used

[0886] 1. Terminal

[0887] Bluetooth earphones and audio glasses are used to capture the user's voice. These devices use a built-in microphone to convert audio input into digital data, which is then sent to a server over the internet.

[0888] 2. Server

[0889] Speech recognition APIs: Used to convert voice data into text data. Examples include Google Cloud Speech-to-Text and Azure Speech Service.

[0890] Natural language processing engines: Used to analyze text data and identify user intent. Examples include OpenAI GPT-3 and Amazon Lex.

[0891] Emotion engines: Used to analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Affectiva.

[0892] Text-to-speech APIs: Used to convert text data into speech data, such as Amazon Polly and Google Cloud Text-to-Speech.

[0893] Specific examples

[0894] Example 1: Checking the schedule

[0895] 1. User: "What's my schedule for tomorrow?"

[0896] 2. Device: Uses microphone to capture audio and send it to the server.

[0897] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's tomorrow's schedule?"

[0898] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is nervous.

[0899] 5. Server: Generate a response using a natural language processing engine (OpenAI GPT-3) saying, "We have a meeting tomorrow at 10:00 AM." Convert this into voice data using a speech synthesis API (Amazon Polly) in a gentle tone.

[0900] 6. Server: Sends audio data to the device.

[0901] 7. Terminal: Play audio data.

[0902] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[0903] Example 2: Checking the weather forecast

[0904] 1. User: "What's the weather like today?"

[0905] 2. Device: Uses microphone to capture audio and send it to the server.

[0906] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's the weather like today?"

[0907] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is calm.

[0908] 5. Server: A natural language processing engine (OpenAI GPT-3) generates a response, stating, "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API (Amazon Polly) in a normal tone.

[0909] 6. Server: Sends audio data to the device.

[0910] 7. Terminal: Play audio data.

[0911] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[0912] This system allows users to obtain information efficiently and naturally without manually operating the device, and to receive responses that correspond to their emotional state at the time. This provides great convenience as it does not require users to interrupt other tasks and can respond in a way that takes into account the user's emotions.

[0913] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0914] Step 1:

[0915] Capturing voice input

[0916] The user puts on Bluetooth earphones or audio glasses and asks questions or gives instructions such as "What's the weather like today?"

[0917] Input: User speech.

[0918] The device uses a built-in microphone to capture the user's speech as audio data, which is then converted into a digital format.

[0919] Output: Audio data in digital format.

[0920] Step 2:

[0921] Sending audio data

[0922] Input: Audio data in digital format.

[0923] The device transmits the captured audio data to a server over a secure network connection.

[0924] Output: The audio data sent to the server.

[0925] Step 3:

[0926] Receiving and decoding audio data

[0927] Input: Audio data sent from the device.

[0928] The server receives the audio data sent from the device. The received audio data is decoded into an analyzable format. This decoding process makes the audio data easier to handle as a digital signal.

[0929] Output: Decoded audio data in a parsable format.

[0930] Step 4:

[0931] Converting audio data to text

[0932] Input: Decoded audio data.

[0933] The server calls a speech recognition API (such as Google Cloud Speech-to-Text) to convert the audio data into text data. During this process, the audio signal is spectrally analyzed and the spoken content is extracted as a string of characters.

[0934] Output: Text data (e.g. "What's the weather like today?").

[0935] Step 5:

[0936] Emotion Analysis

[0937] Input: Text and audio data.

[0938] The server uses an emotion engine (such as IBM Watson Tone Analyzer) to analyze the tone, rate, and intonation of the voice data, thereby recognizing the user's emotional state, such as whether they are nervous, excited, or calm.

[0939] Output: The user's emotional state (e.g., calm).

[0940] Step 6:

[0941] Natural Language Processing and Response Generation

[0942] Input: Text data and sentiment analysis results.

[0943] The server inputs the text data into a natural language processing engine (such as OpenAI GPT-3) to analyze the user's intent. Taking into account the results of sentiment analysis, the server generates an appropriate response and tone. For example, if the user asks, "What's the weather like today?", the response will be, "It's cloudy with occasional rain today, with a maximum temperature of 22 degrees," and if the user is calm, the response should be in a normal tone.

[0944] Output: Response text (e.g., "Today it will be cloudy with occasional rain, with a high of 22 degrees.").

[0945] Step 7:

[0946] Generating audio data

[0947] Input: Response text and sentiment analysis results.

[0948] The server then passes the generated text response to a speech synthesis API (such as Amazon Polly) and converts it into speech data. During speech synthesis, the server also uses the results of emotion analysis to set appropriate intonation and tone.

[0949] Output: Response audio data.

[0950] Step 8:

[0951] Encoding and transmitting audio data

[0952] Input: Response audio data.

[0953] The server encodes the generated voice data for transmission to the terminal, and the encoded voice data is sent back to the terminal via the network.

[0954] Output: The audio data sent to the device.

[0955] Step 9:

[0956] Receiving and playing audio data

[0957] Input: Audio data sent from the server.

[0958] The device receives the audio data sent from the server, decodes it into a playable format, and plays the decoded audio through the device's speaker.

[0959] Output: The audio data to be played (e.g., "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees.").

[0960] (Application example 2)

[0961] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0962] Modern voice interfaces are capable of generating appropriate responses to user inputs. However, these systems are unable to consider the user's emotional state, and the responses are often inappropriate for the user's situation. This results in a poor user experience, as the system is unable to provide intuitively desirable responses based on the user's emotional state. Furthermore, it is difficult to provide personalized suggestions based on the user's emotional state, especially in services such as food delivery.

[0963] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0964] In this invention, the server includes means for capturing voice input, means for converting the voice input from voice to text data, means for recognizing the emotional state of the user, means for generating an appropriate response from the text data taking the emotional state into consideration, means for converting the generated response from text to voice data, and means for transmitting the voice data to the user, thereby enabling personalized content suggestions based on the user's emotional state.

[0965] "Means for capturing audio input" refers to a microphone or sensor that detects the audio emitted by the user and converts it into digital data.

[0966] The "means for transmitting audio input to a server" is a component that has the function of transmitting captured audio data to a remote server via a communication network such as the Internet.

[0967] "Means for converting voice input into text data" refers to software or hardware that uses voice recognition technology to convert received voice data into text.

[0968] "Means for generating an appropriate response" refers to an algorithm or program that analyzes the user's input intent and generates answers or suggestions to the user's questions or requests based on the analysis results.

[0969] The "means for converting a response into voice data" is a system for converting the generated text-format response into voice data using a natural language processing engine or voice synthesis technology.

[0970] The "means for transmitting voice data to a user" refers to a component having a function for transmitting voice data to a user's terminal via a network.

[0971] "Means for reproducing audio data" refers to a speaker or earphone for reproducing audio data received at a user terminal.

[0972] The "means for recognizing emotional state" is a system that includes machine learning models and algorithms for analyzing tone, intonation, facial expressions, etc. from a user's voice data to identify the user's emotional state.

[0973] "Means for generating responses based on emotional state" refers to a program or algorithm for generating responses that take into account the user's perceived emotional state and have a tone and content appropriate to the user's situation.

[0974] A "means for suggesting personalized content based on a user's emotional state" is a system or algorithm for providing appropriate, individualized content or suggestions based on a user's emotional state.

[0975] The present invention is a voice dialogue system including user emotion recognition, particularly adapted for food delivery applications.

[0976] Overall system configuration

[0977] The system mainly consists of the following components:

[0978] 1. A device that captures the user's voice

[0979] 2. Means of transmitting audio data from the device to the server

[0980] 3. Server-side speech recognition, natural language processing, and emotion recognition

[0981] 4. Emotion-based response generation

[0982] 5. A means of converting the response into audio data and sending it to the device

[0983] 6. How to play audio data on the device

[0984] Program processing explanation

[0985] User side (terminal)

[0986] A user inputs voice using a device such as a smartphone or smart glasses. The device captures the voice using a built-in microphone and transmits the voice data to a server via a network.

[0987] Server side

[0988] The server does the following:

[0989] 1. Speech recognition: Use a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert voice data into text data.

[0990] 2. Emotion Recognition: An emotion engine (e.g., cognitive emotion recognition model) analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from text and voice data.

[0991] 3. Natural Language Processing: A natural language processing engine (e.g., OpenAI GPT model) is used to understand the user's intent and generate an appropriate response based on that. The generated response is adjusted in content and tone based on the user's emotional state.

[0992] 4. Convert the response to speech: Use a text-to-speech API (e.g., Amazon Polly) to convert the text response to audio data.

[0993] 5. Sending voice data: The generated voice data is sent again to the user's terminal via the network.

[0994] Actual system usage example

[0995] For example, the system works in the following example:

[0996] What to do if you're tired

[0997] 1. User: "I feel kind of tired..."

[0998] 2. Device: Uses microphone to capture audio and send it to the server.

[0999] 3. Server: Converts the speech into text using a speech recognition API. The server recognizes the phrase "I feel tired."

[1000] 4. Server: The emotion engine recognizes that the user is tired.

[1001] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as, "Cheer up, how about a nutritious salad bowl?"

[1002] 6. Server: Convert into voice data using speech synthesis API.

[1003] 7. Server: Sends audio data to the device.

[1004] 8. Device: Play audio data. The audio says, "Cheer up! How about a nutritious salad bowl?"

[1005] What to do if things are calm

[1006] 1. User: "What should I have for lunch?"

[1007] 2. Device: Uses microphone to capture audio and send it to the server.

[1008] 3. Server: Converts speech to text using a speech recognition API. Recognizes "What should I have for lunch?"

[1009] 4. Server: The emotion engine recognizes that the user is calm.

[1010] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as "You look great today! How about some salmon sushi?"

[1011] 6. Server: Convert into voice data using speech synthesis API.

[1012] 7. Server: Sends audio data to the device.

[1013] 8. Device: Play the audio data. The audio will say, "You look great today! How about some salmon sushi?"

[1014] Prompt Sentence Examples

[1015] Example prompt 1: "Generate a dialogue that suggests a menu to help the user relax when they are tired."

[1016] Example prompt 2: "Generate a dialogue that suggests an energizing meal for the user when they are calm."

[1017] In this way, a system can be implemented that provides personalized responses that take into account the user's emotions.

[1018] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1019] Step 1:

[1020] The user performs voice input. The user issues instructions or asks questions by voice to a device such as a smartphone or smart glasses. The input data is the user's voice (voice signal).

[1021] Step 2:

[1022] The device captures audio input by using the device's built-in microphone to convert the user's voice into digital data, which is raw audio data (such as WAV or MP3 format).

[1023] Step 3:

[1024] The device sends the captured audio data to the server. The digital audio data is sent to the server via a communication network (Wi-Fi or mobile data communication). The input is digital audio data, and the output is the same data transferred to the server.

[1025] Step 4:

[1026] The server converts the received voice data into text using a speech recognition API, such as Google Cloud Speech-to-Text, which analyzes the voice signal and generates a corresponding string of characters. The input is digital voice data, and the output is text instructions or questions.

[1027] Step 5:

[1028] The server inputs the text data into the emotion engine to recognize the user's emotional state. The emotion engine uses a machine learning model to analyze the user's emotions (happiness, sadness, anger, etc.) from the text. The input is text data, and the output is metadata that indicates the user's emotional state.

[1029] Step 6:

[1030] The server uses a natural language processing engine to analyze the user's intent and generate an appropriate response based on that. It also takes into account the results of sentiment analysis to adjust the content and tone of the response. For example, it uses the OpenAI GPT model to generate answers and suggestions to the user's questions. The input is text data and sentiment metadata, and the output is a sentiment-adjusted text response.

[1031] Step 7:

[1032] The server converts the generated text response into audio data using a speech synthesis API. Services such as Amazon Polly are used to synthesize the text into natural-sounding speech. The input is the text response, and the output is audio data (in a playable format).

[1033] Step 8:

[1034] The server encodes the voice data and sends it to the terminal. The generated voice data is then sent back to the user's terminal via the network. The input is the encoded voice data, and the output is the same data transferred to the terminal.

[1035] Step 9:

[1036] The device plays the received audio data. The audio data is played to the user using the device's speaker or earphones. The input is digital audio data, and the output is played back in a form that the user can hear as sound.

[1037] These steps allow users to operate a food delivery application using only their voice and receive personalized suggestions based on their emotions.

[1038] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1039] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1040] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1041] [Third embodiment]

[1042] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1043] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1044] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1045] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1046] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1047] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1048] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1049] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1050] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1051] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1052] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1053] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1054] The present invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. Specific embodiments of this system are described below.

[1055] Overall system configuration

[1056] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[1057] Detailed program processing

[1058] Server Processing

[1059] 1. Receiving voice input

[1060] The server receives the voice data sent by the user and passes it to the voice recognition API.

[1061] 2. Converting voice data to text

[1062] The server converts the voice data into text data using a voice recognition API, which performs, for example, spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[1063] 3. Natural Language Processing and Response Generation

[1064] The server then inputs the resulting text data into a natural language processing engine, which analyzes the voice commands and questions and generates an appropriate response. For example, if a user asks for the weather forecast, the engine queries a weather forecast API and generates a response based on the results.

[1065] 4. Generating Audio Data

[1066] The server then passes the generated text response to a speech synthesis API, which converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[1067] 5. Sending audio data

[1068] The server transmits the generated voice data to the user's terminal via the network.

[1069] Terminal handling

[1070] 1. Capture voice input

[1071] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[1072] 2. Receiving and playing audio data

[1073] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[1074] Specific examples

[1075] Example 1: Checking the schedule

[1076] 1. User: "What's my schedule for tomorrow?"

[1077] 2. Device: Uses microphone to capture audio and send it to the server.

[1078] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[1079] 4. Server: The natural language processing engine calls the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[1080] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1081] 6. Server: Sends audio data to the device.

[1082] 7. Terminal: Play audio data.

[1083] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1084] Example 2: Checking the weather forecast

[1085] 1. User: "What's the weather like today?"

[1086] 2. Device: Uses microphone to capture audio and send it to the server.

[1087] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[1088] 4. Server: The natural language processing engine calls the weather forecast API and generates a response: "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[1089] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1090] 6. Server: Sends audio data to the device.

[1091] 7. Terminal: Play audio data.

[1092] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1093] This system allows users to efficiently obtain information by voice without manual input or visual confirmation, providing great convenience especially when hands are busy or during tasks that require visual concentration.

[1094] The processing flow will be explained below.

[1095] Step 1:

[1096] The user inputs voice into the Bluetooth earphones or audio glasses. The user's speech is a question or instruction such as "Tell me today's weather."

[1097] Step 2:

[1098] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[1099] Step 3:

[1100] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[1101] Step 4:

[1102] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[1103] Step 5:

[1104] The server passes the text data to a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. For example, it may determine that the request is for a weather forecast. The natural language processing engine analyzes the input text data and determines an appropriate response.

[1105] Step 6:

[1106] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, the server generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1107] Step 7:

[1108] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[1109] Step 8:

[1110] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the encoded voice data to the terminal via the network.

[1111] Step 9:

[1112] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[1113] Step 10:

[1114] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1115] This process flow allows users to obtain information and execute instructions using only their voice, without having to manually operate the device. This allows users to solve problems efficiently without interrupting other tasks.

[1116] Example 1

[1117] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1118] Conventional voice interface systems often require users to manually input information or look at a screen when obtaining it through voice, limiting their convenience. Furthermore, these systems are only effective in specific locations and situations, making flexible use difficult. Furthermore, if the accuracy of speech recognition and response generation is low, users may not receive satisfactory service.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1120] In this invention, the server includes means for acquiring voice input from a user, means for transmitting the voice input to a processing device, means for converting the voice input into text data, means for generating an appropriate response based on the text data, means for converting the response into voice data, means for transmitting the voice data to the user, and means for playing back the voice data. This allows the user to efficiently acquire information in a natural, interactive format without having to manually input or check a screen. Specifically, by using a voice recognition API and a natural language processing engine, the server can analyze the user's intent with high accuracy and generate and provide an appropriate response.

[1121] A "means for obtaining voice input" is a device that senses a user's speech and converts it into a voice signal.

[1122] The "means for transmitting to a processing unit" is a device or function that transmits the captured audio signal as digital data to a central processing unit via a network.

[1123] "Means for converting into text data" refers to a function or device that analyzes a voice signal and converts it into text data.

[1124] The "means for generating an appropriate response" refers to a function or device that analyzes the user's intention based on character data and generates a response in natural language.

[1125] The "means for converting into voice data" refers to a function or device that converts the generated text response into voice data.

[1126] The "means for transmitting to the user" refers to a function or device that transmits voice data to the user's terminal via a network.

[1127] The "means for reproducing audio data" is a device such as a speaker that reproduces audio data received by the user's terminal.

[1128] The "means for analyzing in natural language" refers to a function or device for analyzing acquired text data and understanding the intent of the user's questions and instructions.

[1129] An "external language processing engine" is an engine with natural language processing capabilities that is provided through a cloud service or external API.

[1130] This invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connection, and receive responses by voice. Here we will show how to specifically implement this system.

[1131] Overall system configuration

[1132] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[1133] Hardware and software used

[1134] Devices: Bluetooth earphones, audio glasses with built-in mobile connection

[1135] Server: High-performance cloud computing services

[1136] Speech recognition API: Google Speech-to-Text, etc.

[1137] Natural language processing engine: OpenAI GPT-3, etc.

[1138] Speech synthesis API: Google Text-to-Speech, etc.

[1139] Network: Internet connection (Wi-Fi, mobile data)

[1140] Processing of various data

[1141] Receiving and processing voice input

[1142] User: Ask questions and give instructions using voice, just like in everyday conversation.

[1143] The device captures the user's voice using a built-in microphone, and the voice data is compressed through digital signal processing.

[1144] The terminal transmits the compressed audio data to the server via the network.

[1145] Speech-to-text transcription and analysis

[1146] The server receives the audio data over the network.

[1147] The server converts the received voice data into text using the Google Speech-to-Text API.

[1148] The server passes the converted text data to a natural language processing engine such as OpenAI GPT-3, which analyzes the user's intent and generates an appropriate response.

[1149] Response generation and speech synthesis

[1150] The server passes the generated text response to a speech synthesis API, such as the Google Text-to-Speech API, and converts it into audio data.

[1151] The server encodes the generated voice data and transmits it to the user's terminal.

[1152] Playing audio

[1153] The terminal decodes the received audio data and plays it on the speaker.

[1154] The user hears the response from the server through the played audio.

[1155] Specific examples

[1156] Example 1: Checking the schedule

[1157] 1. User: "What's my schedule for tomorrow?"

[1158] 2. Device: Uses microphone to capture audio and send it to the server.

[1159] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[1160] 4. Server: The natural language processing engine queries the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[1161] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1162] 6. Server: Sends audio data to the device.

[1163] 7. Terminal: Play audio data.

[1164] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1165] Example 2: Checking the weather forecast

[1166] 1. User: "What's the weather like today?"

[1167] 2. Device: Uses microphone to capture audio and send it to the server.

[1168] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[1169] 4. Server: The natural language processing engine queries the weather API and generates a response: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[1170] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1171] 6. Server: Sends audio data to the device.

[1172] 7. Terminal: Play audio data.

[1173] 8. User: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[1174] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1175] Step 1:

[1176] The user asks questions or gives instructions by voice.

[1177] Input: User utterance "What's the weather like today?"

[1178] Specific actions: The user speaks through Bluetooth earphones or audio glasses.

[1179] Step 2:

[1180] The terminal captures the user's speech.

[1181] Input: User's spoken utterance

[1182] What it does: It uses a built-in microphone to capture the user's speech as digital audio data.

[1183] Output: Digital audio data

[1184] Step 3:

[1185] The terminal transmits the captured audio data to the server.

[1186] Input: Digital audio data

[1187] What it does: It compresses digital audio data and sends it over the network to a server.

[1188] Output: Compressed audio data

[1189] Step 4:

[1190] The server passes the received voice data to the voice recognition API.

[1191] Input: Compressed audio data

[1192] Specific operation: Decodes the audio data and passes it to a speech recognition API such as the Google Speech-to-Text API.

[1193] Output: Text data "What's the weather like today?"

[1194] Step 5:

[1195] The server inputs the data converted into text by the speech recognition API into a natural language processing engine.

[1196] Input: Text data "What's the weather like today?"

[1197] What it does: Provides text data as prompts to a natural language processing engine such as OpenAI GPT-3.

[1198] Output: Response "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1199] Step 6:

[1200] The server passes the generated response text to the speech synthesis API.

[1201] Input: Response "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[1202] What it does: Passes the response text to a speech synthesis API, such as the Google Text-to-Speech API, to generate natural-sounding speech.

[1203] Output: Audio data

[1204] Step 7:

[1205] The server transmits the generated voice data to the user's terminal.

[1206] Input: Audio data

[1207] Specific operations: Encodes audio data and sends it over the network to the user's device.

[1208] Output: Audio data

[1209] Step 8:

[1210] The terminal decodes the received audio data and plays it on the speaker.

[1211] Input: Received audio data

[1212] Specific behavior: Decodes audio data and plays it to the user through the built-in speaker.

[1213] Output: Audio played: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[1214] (Application example 1)

[1215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1216] Current voice interfaces for autonomous vehicles have difficulty accurately analyzing user voice input and providing appropriate information. Furthermore, there is a lack of technology that allows drivers to efficiently operate autonomous vehicles through voice commands, making it difficult to balance safe operation with information provision. Therefore, there is a demand for a system that can improve the accuracy of voice interfaces within autonomous vehicles and provide appropriate information needed during operation.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1218] In this invention, the server includes means for capturing voice input from a user, means for transmitting the voice input to the server, and means for converting the voice input into text data, thereby enabling generation of appropriate route guidance based on position information within the autonomous mobile body.

[1219] "Voice input" is voice data uttered by the user through a microphone.

[1220] A "server" is a computer system that analyzes voice and text data over a network and generates an appropriate response.

[1221] "Text data" is character information converted by speech recognition.

[1222] A "response" is information or instructions generated based on voice input.

[1223] "Audio data" is data in an audio format that is converted from text data by voice synthesis.

[1224] An "autonomous mobile object" is a machine such as a vehicle or robot that moves autonomously without human operation.

[1225] "Route guidance" refers to instructions that include route information to a destination.

[1226] "Means for analyzing in natural language" refers to methods and technologies for analyzing and understanding voice input using language processing technology.

[1227] A "map information processing engine" is a system that analyzes map data and provides route guidance and identifies the current location.

[1228] Overall system configuration

[1229] This system consists of a voice interface device installed in the autonomous vehicle and an external cloud server. Users can communicate with the autonomous vehicle through the voice interface and receive various information and instructions.

[1230] Hardware Configuration

[1231] Voice interface device for autonomous mobile body

[1232] The audio interface device includes the following hardware:

[1233] Microphone: Captures audio input.

[1234] Speaker: Plays back the generated audio data.

[1235] Communication module: Sends and receives data to and from the server.

[1236] server

[1237] It exists on the cloud and has the following functions:

[1238] Speech Recognition API: Converts voice data into text.

[1239] Natural language processing engine: Analyzes text and generates responses.

[1240] Map information processing engine: Generates route guidance based on location information.

[1241] Speech synthesis API: Converts text into audio data.

[1242] Software configuration and data processing

[1243] 1. Capture voice input

[1244] When a user speaks into the voice interface device, a built-in microphone captures the voice and transmits the data to a server.

[1245] 2. Converting voice data to text

[1246] The server converts the transmitted voice data into text data using a speech recognition API, which performs spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[1247] 3. Natural Language Processing and Response Generation

[1248] The text data is fed into a natural language processing engine, which analyzes the question or instruction and generates an appropriate response. For example, if the user requests route guidance, a map information processing engine is called to calculate the optimal route.

[1249] 4. Generate response audio data

[1250] The generated response text is passed to a speech synthesis API and converted into audio data, which takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[1251] 5. Sending and Playing Audio Data

[1252] The generated voice data is transmitted to the voice interface device through the communication module of the autonomous mobile body and played back through a speaker.

[1253] Specific examples

[1254] 1. Providing route guidance

[1255] User: "What's the quickest route to my house?"

[1256] Server: Converts speech to text using speech recognition API.

[1257] Server: The natural language processing engine analyzes the request "Tell me the shortest route to my house" and calls the map information processing engine to obtain route information.

[1258] Server: Convert the generated route guidance into voice data using a speech synthesis API.

[1259] Autonomous mobile unit: Receives and plays back audio data.

[1260] 2. Examples of prompts

[1261] "Please tell me the shortest route to my destination. Please specify the fastest route. Starting point: current location, destination: home. Please also take current traffic information into consideration."

[1262] This system allows users to efficiently obtain information by voice without operating the vehicle, improving safety while driving.

[1263] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1264] Step 1:

[1265] Capture voice input

[1266] Input: User utterance

[1267] Output: Audio data

[1268] Specific operation: When a user speaks to the voice interface device, the terminal (interface device in the autonomous mobile body) captures the voice using a microphone. The captured voice is sent as digital data to the server via the communication module.

[1269] Step 2:

[1270] Converting audio data to text

[1271] Input: Audio data

[1272] Output: Text data

[1273] Specific operation: The server inputs the received voice data into the voice recognition API. The voice recognition API performs spectral analysis of the voice signal and generates corresponding text data. For example, if a user says, "Tell me the shortest route to my house," the voice data is converted into text data saying, "Tell me the shortest route to my house."

[1274] Step 3:

[1275] Natural Language Processing and Response Generation

[1276] Input: Text data

[1277] Output: Response text

[1278] Specific operation: The server inputs the converted text data into a natural language processing engine. The natural language processing engine analyzes the text data and understands the user's instructions and questions. For example, in response to an input such as "What is the shortest route to my house?", the engine calls a map information processing engine and generates appropriate route guidance. The generated response text is "The shortest route from my current location to my house is..."

[1279] Step 4:

[1280] Generate response audio data

[1281] Input: Response text

[1282] Output: Audio data

[1283] Specific operation: The server passes the generated response text to the speech synthesis API. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech data. For example, a text response such as "The shortest route from your current location to your home is..." is converted into speech data.

[1284] Step 5:

[1285] Sending and playing audio data

[1286] Input: Audio data

[1287] Output: Audio output to the user

[1288] Specific operation: The generated voice data is sent from the server to the voice interface device via the communication module of the autonomous mobile device. The terminal plays the received voice data through a speaker and provides a response to the user. For example, the user can hear a voice response such as, "The shortest route from your current location to your home is..."

[1289] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1290] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[1291] Overall system configuration

[1292] This system consists of a user device (such as Bluetooth earphones or audio glasses), a server located in the cloud, and an emotion engine. The user asks questions or gives instructions via voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[1293] Detailed program processing

[1294] Server Processing

[1295] 1. Receiving voice input

[1296] The server receives the voice data sent from the device, which is first decoded and converted into an analyzable format.

[1297] 2. Converting voice data to text

[1298] The server converts the voice data into text data using a speech recognition API, performs spectral analysis of the voice signal, and extracts the spoken content as a string of characters.

[1299] 3. Emotion Analysis

[1300] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[1301] 4. Natural Language Processing and Response Generation

[1302] The server inputs the text data into a natural language processing engine to analyze the user's intent. It also takes into account the results of sentiment analysis and adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[1303] 5. Generating Audio Data

[1304] The server then passes the generated text response to a speech synthesis API, which converts it into audio data, adjusting intonation and tone based on the results of sentiment analysis.

[1305] 6. Sending audio data

[1306] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[1307] Terminal handling

[1308] 1. Capture voice input

[1309] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[1310] 2. Receiving and playing audio data

[1311] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[1312] Specific examples

[1313] Example 1: Checking the schedule

[1314] 1. User: "What's my schedule for tomorrow?"

[1315] 2. Device: Uses microphone to capture audio and send it to the server.

[1316] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[1317] 4. Server: The emotion engine recognizes that the user is nervous.

[1318] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10:00 AM." This is converted into voice data using a speech synthesis API in a gentle tone.

[1319] 6. Server: Sends audio data to the device.

[1320] 7. Terminal: Play audio data.

[1321] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1322] Example 2: Checking the weather forecast

[1323] 1. User: "What's the weather like today?"

[1324] 2. Device: Uses microphone to capture audio and send it to the server.

[1325] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[1326] 4. Server: The emotion engine recognizes that the user is calm.

[1327] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[1328] 6. Server: Sends audio data to the device.

[1329] 7. Terminal: Play audio data.

[1330] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1331] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[1332] The processing flow will be explained below.

[1333] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[1334] Detailed program processing

[1335] Step 1:

[1336] The user speaks into the Bluetooth earphones or audio glasses to ask questions or give instructions, such as "Tell me the weather today."

[1337] Step 2:

[1338] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[1339] Step 3:

[1340] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[1341] Step 4:

[1342] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[1343] Step 5:

[1344] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[1345] Step 6:

[1346] The server inputs the text data into a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. Taking into account the results of sentiment analysis, the server adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[1347] Step 7:

[1348] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, it generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1349] Step 8:

[1350] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API generates natural-sounding speech by taking into account the grammatical structure and intonation of the text. It also adjusts the intonation and tone based on the results of sentiment analysis.

[1351] Step 9:

[1352] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[1353] Step 10:

[1354] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[1355] Step 11:

[1356] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1357] Specific examples

[1358] Example 1: Checking the schedule

[1359] 1. User: "What's my schedule for tomorrow?"

[1360] 2. Device: Uses microphone to capture audio and send it to the server.

[1361] 3. Server: Use a speech recognition API to convert the speech into text, saying "What's my schedule for tomorrow?"

[1362] 4. Server: The emotion engine recognizes that the user is nervous.

[1363] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10 AM." The response is converted into voice data using a speech synthesis API in a gentle tone.

[1364] 6. Server: Sends the audio data to the device.

[1365] 7. Terminal: Plays audio data.

[1366] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1367] Example 2: Checking the weather forecast

[1368] 1. User: "What's the weather like today?"

[1369] 2. Device: Uses microphone to capture audio and send it to the server.

[1370] 3. Server: Use the speech recognition API to convert the speech into text, saying "What's the weather like today?"

[1371] 4. Server: The emotion engine recognizes that the user is calm.

[1372] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[1373] 6. Server: Sends the audio data to the device.

[1374] 7. Terminal: Plays audio data.

[1375] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1376] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[1377] Example 2

[1378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1379] While modern voice assistant systems typically provide responses based on user utterances, they rarely consider the user's emotional state. As a result, appropriate responses are not provided based on the user's emotional state, resulting in a poor user experience. In addition, the system's inability to recognize the user's emotions makes it difficult to provide more human-like interactions.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1381] In this invention, the server includes a means for converting voice input into text data, a means for recognizing the user's emotions based on the text data, and a means for analyzing the user's intentions and generating an appropriate response according to the emotions, thereby enabling a more natural and human-like response that takes the user's emotions into consideration.

[1382] A "user" is a person who uses the system to provide voice input and receive responses.

[1383] "Voice input" refers to a voice signal of a question or instruction verbally given by the user.

[1384] A "capturing means" is a device or method for converting a user's voice input into digital data and acquiring it.

[1385] A "server" is a central object that works in conjunction with terminals via a network to analyze voice data, convert it into text, analyze emotions, and generate responses.

[1386] "Text data" is voice data converted into a character string, and is generated by voice recognition.

[1387] A "converting means" is a technique or device used to convert voice data into text data or text data into voice data.

[1388] "Means for recognizing emotions" refers to technologies and algorithms that analyze text data and voice data to determine a user's emotional state.

[1389] "Means for analyzing intent" refers to techniques and methods for understanding what a user wants based on the content of their speech and their emotional state, and generating an appropriate response.

[1390] "Means for generating a response" refers to techniques and methods for generating appropriate information or answers in response to the analyzed user's intentions and emotions.

[1391] "Means for reproducing audio data" refers to a device or method for providing the generated audio data to the user in a reproducible format.

[1392] The present invention aims to develop a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connections, and receive responses by voice. By further combining this with an emotion engine, the system can recognize the user's emotions and generate more appropriate responses.

[1393] This system consists of a user's device and a server and emotion engine located on the cloud. The user speaks questions or commands through the device, and the voice data is sent to the server via the network. The server converts the voice data into text data, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays back the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[1394] Hardware and software used

[1395] 1. Terminal

[1396] Bluetooth earphones and audio glasses are used to capture the user's voice. These devices use a built-in microphone to convert audio input into digital data, which is then sent to a server over the internet.

[1397] 2. Server

[1398] Speech recognition APIs: Used to convert voice data into text data. Examples include Google Cloud Speech-to-Text and Azure Speech Service.

[1399] Natural language processing engines: Used to analyze text data and identify user intent. Examples include OpenAI GPT-3 and Amazon Lex.

[1400] Emotion engines: Used to analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Affectiva.

[1401] Text-to-speech APIs: Used to convert text data into speech data, such as Amazon Polly and Google Cloud Text-to-Speech.

[1402] Specific examples

[1403] Example 1: Checking the schedule

[1404] 1. User: "What's my schedule for tomorrow?"

[1405] 2. Device: Uses microphone to capture audio and send it to the server.

[1406] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's tomorrow's schedule?"

[1407] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is nervous.

[1408] 5. Server: Generate a response using a natural language processing engine (OpenAI GPT-3) saying, "We have a meeting tomorrow at 10:00 AM." Convert this into voice data using a speech synthesis API (Amazon Polly) in a gentle tone.

[1409] 6. Server: Sends audio data to the device.

[1410] 7. Terminal: Play audio data.

[1411] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1412] Example 2: Checking the weather forecast

[1413] 1. User: "What's the weather like today?"

[1414] 2. Device: Uses microphone to capture audio and send it to the server.

[1415] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's the weather like today?"

[1416] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is calm.

[1417] 5. Server: A natural language processing engine (OpenAI GPT-3) generates a response, stating, "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API (Amazon Polly) in a normal tone.

[1418] 6. Server: Sends audio data to the device.

[1419] 7. Terminal: Play audio data.

[1420] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1421] This system allows users to obtain information efficiently and naturally without manually operating the device, and to receive responses that correspond to their emotional state at the time. This provides great convenience as it does not require users to interrupt other tasks and can respond in a way that takes into account the user's emotions.

[1422] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1423] Step 1:

[1424] Capturing voice input

[1425] The user puts on Bluetooth earphones or audio glasses and asks questions or gives instructions such as "What's the weather like today?"

[1426] Input: User speech.

[1427] The device uses a built-in microphone to capture the user's speech as audio data, which is then converted into a digital format.

[1428] Output: Audio data in digital format.

[1429] Step 2:

[1430] Sending audio data

[1431] Input: Audio data in digital format.

[1432] The device transmits the captured audio data to a server over a secure network connection.

[1433] Output: The audio data sent to the server.

[1434] Step 3:

[1435] Receiving and decoding audio data

[1436] Input: Audio data sent from the device.

[1437] The server receives the audio data sent from the device. The received audio data is decoded into an analyzable format. This decoding process makes the audio data easier to handle as a digital signal.

[1438] Output: Decoded audio data in a parsable format.

[1439] Step 4:

[1440] Converting audio data to text

[1441] Input: Decoded audio data.

[1442] The server calls a speech recognition API (such as Google Cloud Speech-to-Text) to convert the audio data into text data. During this process, the audio signal is spectrally analyzed and the spoken content is extracted as a string of characters.

[1443] Output: Text data (e.g. "What's the weather like today?").

[1444] Step 5:

[1445] Emotion Analysis

[1446] Input: Text and audio data.

[1447] The server uses an emotion engine (such as IBM Watson Tone Analyzer) to analyze the tone, rate, and intonation of the voice data, thereby recognizing the user's emotional state, such as whether they are nervous, excited, or calm.

[1448] Output: The user's emotional state (e.g., calm).

[1449] Step 6:

[1450] Natural Language Processing and Response Generation

[1451] Input: Text data and sentiment analysis results.

[1452] The server inputs the text data into a natural language processing engine (such as OpenAI GPT-3) to analyze the user's intent. Taking into account the results of sentiment analysis, the server generates an appropriate response and tone. For example, if the user asks, "What's the weather like today?", the response will be, "It's cloudy with occasional rain today, with a maximum temperature of 22 degrees," and if the user is calm, the response should be in a normal tone.

[1453] Output: Response text (e.g., "Today it will be cloudy with occasional rain, with a high of 22 degrees.").

[1454] Step 7:

[1455] Generating audio data

[1456] Input: Response text and sentiment analysis results.

[1457] The server then passes the generated text response to a speech synthesis API (such as Amazon Polly) and converts it into speech data. During speech synthesis, the server also uses the results of emotion analysis to set appropriate intonation and tone.

[1458] Output: Response audio data.

[1459] Step 8:

[1460] Encoding and transmitting audio data

[1461] Input: Response audio data.

[1462] The server encodes the generated voice data for transmission to the terminal, and the encoded voice data is sent back to the terminal via the network.

[1463] Output: The audio data sent to the device.

[1464] Step 9:

[1465] Receiving and playing audio data

[1466] Input: Audio data sent from the server.

[1467] The device receives the audio data sent from the server, decodes it into a playable format, and plays the decoded audio through the device's speaker.

[1468] Output: The audio data to be played (e.g., "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees.").

[1469] (Application example 2)

[1470] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1471] Modern voice interfaces are capable of generating appropriate responses to user inputs. However, these systems are unable to consider the user's emotional state, and the responses are often inappropriate for the user's situation. This results in a poor user experience, as the system is unable to provide intuitively desirable responses based on the user's emotional state. Furthermore, it is difficult to provide personalized suggestions based on the user's emotional state, especially in services such as food delivery.

[1472] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1473] In this invention, the server includes means for capturing voice input, means for converting the voice input from voice to text data, means for recognizing the emotional state of the user, means for generating an appropriate response from the text data taking the emotional state into consideration, means for converting the generated response from text to voice data, and means for transmitting the voice data to the user, thereby enabling personalized content suggestions based on the user's emotional state.

[1474] "Means for capturing audio input" refers to a microphone or sensor that detects the audio emitted by the user and converts it into digital data.

[1475] The "means for transmitting audio input to a server" is a component that has the function of transmitting captured audio data to a remote server via a communication network such as the Internet.

[1476] "Means for converting voice input into text data" refers to software or hardware that uses voice recognition technology to convert received voice data into text.

[1477] "Means for generating an appropriate response" refers to an algorithm or program that analyzes the user's input intent and generates answers or suggestions to the user's questions or requests based on the analysis results.

[1478] The "means for converting a response into voice data" is a system for converting the generated text-format response into voice data using a natural language processing engine or voice synthesis technology.

[1479] The "means for transmitting voice data to a user" refers to a component having a function for transmitting voice data to a user's terminal via a network.

[1480] "Means for reproducing audio data" refers to a speaker or earphone for reproducing audio data received at a user terminal.

[1481] The "means for recognizing emotional state" is a system that includes machine learning models and algorithms for analyzing tone, intonation, facial expressions, etc. from a user's voice data to identify the user's emotional state.

[1482] "Means for generating responses based on emotional state" refers to a program or algorithm for generating responses that take into account the user's perceived emotional state and have a tone and content appropriate to the user's situation.

[1483] A "means for suggesting personalized content based on a user's emotional state" is a system or algorithm for providing appropriate, individualized content or suggestions based on a user's emotional state.

[1484] The present invention is a voice dialogue system including user emotion recognition, particularly adapted for food delivery applications.

[1485] Overall system configuration

[1486] The system mainly consists of the following components:

[1487] 1. A device that captures the user's voice

[1488] 2. Means of transmitting audio data from the device to the server

[1489] 3. Server-side speech recognition, natural language processing, and emotion recognition

[1490] 4. Emotion-based response generation

[1491] 5. A means of converting the response into audio data and sending it to the device

[1492] 6. How to play audio data on the device

[1493] Program processing explanation

[1494] User side (terminal)

[1495] A user inputs voice using a device such as a smartphone or smart glasses. The device captures the voice using a built-in microphone and transmits the voice data to a server via a network.

[1496] Server side

[1497] The server does the following:

[1498] 1. Speech recognition: Use a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert voice data into text data.

[1499] 2. Emotion Recognition: An emotion engine (e.g., cognitive emotion recognition model) analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from text and voice data.

[1500] 3. Natural Language Processing: A natural language processing engine (e.g., OpenAI GPT model) is used to understand the user's intent and generate an appropriate response based on that. The generated response is adjusted in content and tone based on the user's emotional state.

[1501] 4. Convert the response to speech: Use a text-to-speech API (e.g., Amazon Polly) to convert the text response to audio data.

[1502] 5. Sending voice data: The generated voice data is sent again to the user's terminal via the network.

[1503] Actual system usage example

[1504] For example, the system works in the following example:

[1505] What to do if you're tired

[1506] 1. User: "I feel kind of tired..."

[1507] 2. Device: Uses microphone to capture audio and send it to the server.

[1508] 3. Server: Converts the speech into text using a speech recognition API. The server recognizes the phrase "I feel tired."

[1509] 4. Server: The emotion engine recognizes that the user is tired.

[1510] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as, "Cheer up, how about a nutritious salad bowl?"

[1511] 6. Server: Convert into voice data using speech synthesis API.

[1512] 7. Server: Sends audio data to the device.

[1513] 8. Device: Play audio data. The audio says, "Cheer up! How about a nutritious salad bowl?"

[1514] What to do if things are calm

[1515] 1. User: "What should I have for lunch?"

[1516] 2. Device: Uses microphone to capture audio and send it to the server.

[1517] 3. Server: Converts speech to text using a speech recognition API. Recognizes "What should I have for lunch?"

[1518] 4. Server: The emotion engine recognizes that the user is calm.

[1519] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as "You look great today! How about some salmon sushi?"

[1520] 6. Server: Convert into voice data using speech synthesis API.

[1521] 7. Server: Sends audio data to the device.

[1522] 8. Device: Play the audio data. The audio will say, "You look great today! How about some salmon sushi?"

[1523] Prompt Sentence Examples

[1524] Example prompt 1: "Generate a dialogue that suggests a menu to help the user relax when they are tired."

[1525] Example prompt 2: "Generate a dialogue that suggests an energizing meal for the user when they are calm."

[1526] In this way, a system can be implemented that provides personalized responses that take into account the user's emotions.

[1527] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1528] Step 1:

[1529] The user performs voice input. The user issues instructions or asks questions by voice to a device such as a smartphone or smart glasses. The input data is the user's voice (voice signal).

[1530] Step 2:

[1531] The device captures audio input by using the device's built-in microphone to convert the user's voice into digital data, which is raw audio data (such as WAV or MP3 format).

[1532] Step 3:

[1533] The device sends the captured audio data to the server. The digital audio data is sent to the server via a communication network (Wi-Fi or mobile data communication). The input is digital audio data, and the output is the same data transferred to the server.

[1534] Step 4:

[1535] The server converts the received voice data into text using a speech recognition API, such as Google Cloud Speech-to-Text, which analyzes the voice signal and generates a corresponding string of characters. The input is digital voice data, and the output is text instructions or questions.

[1536] Step 5:

[1537] The server inputs the text data into the emotion engine to recognize the user's emotional state. The emotion engine uses a machine learning model to analyze the user's emotions (happiness, sadness, anger, etc.) from the text. The input is text data, and the output is metadata that indicates the user's emotional state.

[1538] Step 6:

[1539] The server uses a natural language processing engine to analyze the user's intent and generate an appropriate response based on that. It also takes into account the results of sentiment analysis to adjust the content and tone of the response. For example, it uses the OpenAI GPT model to generate answers and suggestions to the user's questions. The input is text data and sentiment metadata, and the output is a sentiment-adjusted text response.

[1540] Step 7:

[1541] The server converts the generated text response into audio data using a speech synthesis API. Services such as Amazon Polly are used to synthesize the text into natural-sounding speech. The input is the text response, and the output is audio data (in a playable format).

[1542] Step 8:

[1543] The server encodes the voice data and sends it to the terminal. The generated voice data is then sent back to the user's terminal via the network. The input is the encoded voice data, and the output is the same data transferred to the terminal.

[1544] Step 9:

[1545] The device plays the received audio data. The audio data is played to the user using the device's speaker or earphones. The input is digital audio data, and the output is played back in a form that the user can hear as sound.

[1546] These steps allow users to operate a food delivery application using only their voice and receive personalized suggestions based on their emotions.

[1547] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1548] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1549] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1550] [Fourth embodiment]

[1551] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1552] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1553] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1554] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1555] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1556] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1557] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1558] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1559] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1560] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1561] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1562] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1563] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1564] The present invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. Specific embodiments of this system are described below.

[1565] Overall system configuration

[1566] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[1567] Detailed program processing

[1568] Server Processing

[1569] 1. Receiving voice input

[1570] The server receives the voice data sent by the user and passes it to the voice recognition API.

[1571] 2. Converting voice data to text

[1572] The server converts the voice data into text data using a voice recognition API, which performs, for example, spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[1573] 3. Natural Language Processing and Response Generation

[1574] The server then inputs the resulting text data into a natural language processing engine, which analyzes the voice commands and questions and generates an appropriate response. For example, if a user asks for the weather forecast, the engine queries a weather forecast API and generates a response based on the results.

[1575] 4. Generating Audio Data

[1576] The server then passes the generated text response to a speech synthesis API, which converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[1577] 5. Sending audio data

[1578] The server transmits the generated voice data to the user's terminal via the network.

[1579] Terminal handling

[1580] 1. Capture voice input

[1581] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[1582] 2. Receiving and playing audio data

[1583] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[1584] Specific examples

[1585] Example 1: Checking the schedule

[1586] 1. User: "What's my schedule for tomorrow?"

[1587] 2. Device: Uses microphone to capture audio and send it to the server.

[1588] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[1589] 4. Server: The natural language processing engine calls the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[1590] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1591] 6. Server: Sends audio data to the device.

[1592] 7. Terminal: Play audio data.

[1593] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1594] Example 2: Checking the weather forecast

[1595] 1. User: "What's the weather like today?"

[1596] 2. Device: Uses microphone to capture audio and send it to the server.

[1597] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[1598] 4. Server: The natural language processing engine calls the weather forecast API and generates a response: "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[1599] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1600] 6. Server: Sends audio data to the device.

[1601] 7. Terminal: Play audio data.

[1602] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1603] This system allows users to efficiently obtain information by voice without manual input or visual confirmation, providing great convenience especially when hands are busy or during tasks that require visual concentration.

[1604] The processing flow will be explained below.

[1605] Step 1:

[1606] The user inputs voice into the Bluetooth earphones or audio glasses. The user's speech is a question or instruction such as "Tell me today's weather."

[1607] Step 2:

[1608] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[1609] Step 3:

[1610] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[1611] Step 4:

[1612] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[1613] Step 5:

[1614] The server passes the text data to a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. For example, it may determine that the request is for a weather forecast. The natural language processing engine analyzes the input text data and determines an appropriate response.

[1615] Step 6:

[1616] The server sends a query to the weather forecast API to get current weather information. The response from the API includes information such as the weather and temperature. Based on this response, the server generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1617] Step 7:

[1618] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[1619] Step 8:

[1620] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the encoded voice data to the terminal via the network.

[1621] Step 9:

[1622] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[1623] Step 10:

[1624] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1625] This process flow allows users to obtain information and execute instructions using only their voice, without having to manually operate the device. This allows users to solve problems efficiently without interrupting other tasks.

[1626] Example 1

[1627] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1628] Conventional voice interface systems often require users to manually input information or look at a screen when obtaining it through voice, limiting their convenience. Furthermore, these systems are only effective in specific locations and situations, making flexible use difficult. Furthermore, if the accuracy of speech recognition and response generation is low, users may not receive satisfactory service.

[1629] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1630] In this invention, the server includes means for acquiring voice input from a user, means for transmitting the voice input to a processing device, means for converting the voice input into text data, means for generating an appropriate response based on the text data, means for converting the response into voice data, means for transmitting the voice data to the user, and means for playing back the voice data. This allows the user to efficiently acquire information in a natural, interactive format without having to manually input or check a screen. Specifically, by using a voice recognition API and a natural language processing engine, the server can analyze the user's intent with high accuracy and generate and provide an appropriate response.

[1631] A "means for obtaining voice input" is a device that senses a user's speech and converts it into a voice signal.

[1632] The "means for transmitting to a processing unit" is a device or function that transmits the captured audio signal as digital data to a central processing unit via a network.

[1633] "Means for converting into text data" refers to a function or device that analyzes a voice signal and converts it into text data.

[1634] The "means for generating an appropriate response" refers to a function or device that analyzes the user's intention based on character data and generates a response in natural language.

[1635] The "means for converting into voice data" refers to a function or device that converts the generated text response into voice data.

[1636] The "means for transmitting to the user" refers to a function or device that transmits voice data to the user's terminal via a network.

[1637] The "means for reproducing audio data" is a device such as a speaker that reproduces audio data received by the user's terminal.

[1638] The "means for analyzing in natural language" refers to a function or device for analyzing acquired text data and understanding the intent of the user's questions and instructions.

[1639] An "external language processing engine" is an engine with natural language processing capabilities that is provided through a cloud service or external API.

[1640] This invention is a system that allows users to ask questions or give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connection, and receive responses by voice. Here we will show how to specifically implement this system.

[1641] Overall system configuration

[1642] This system consists of a user device (such as Bluetooth earphones or audio glasses) and a server located on the cloud. The user asks questions or gives instructions by voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data so that the user can listen to it.

[1643] Hardware and software used

[1644] Devices: Bluetooth earphones, audio glasses with built-in mobile connection

[1645] Server: High-performance cloud computing services

[1646] Speech recognition API: Google Speech-to-Text, etc.

[1647] Natural language processing engine: OpenAI GPT-3, etc.

[1648] Speech synthesis API: Google Text-to-Speech, etc.

[1649] Network: Internet connection (Wi-Fi, mobile data)

[1650] Processing of various data

[1651] Receiving and processing voice input

[1652] User: Ask questions and give instructions using voice, just like in everyday conversation.

[1653] The device captures the user's voice using a built-in microphone, and the voice data is compressed through digital signal processing.

[1654] The terminal transmits the compressed audio data to the server via the network.

[1655] Speech-to-text transcription and analysis

[1656] The server receives the audio data over the network.

[1657] The server converts the received voice data into text using the Google Speech-to-Text API.

[1658] The server passes the converted text data to a natural language processing engine such as OpenAI GPT-3, which analyzes the user's intent and generates an appropriate response.

[1659] Response generation and speech synthesis

[1660] The server passes the generated text response to a speech synthesis API, such as the Google Text-to-Speech API, and converts it into audio data.

[1661] The server encodes the generated voice data and transmits it to the user's terminal.

[1662] Playing audio

[1663] The terminal decodes the received audio data and plays it on the speaker.

[1664] The user hears the response from the server through the played audio.

[1665] Specific examples

[1666] Example 1: Checking the schedule

[1667] 1. User: "What's my schedule for tomorrow?"

[1668] 2. Device: Uses microphone to capture audio and send it to the server.

[1669] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[1670] 4. Server: The natural language processing engine queries the schedule management API and generates a response, such as "There is a meeting tomorrow at 10 AM."

[1671] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1672] 6. Server: Sends audio data to the device.

[1673] 7. Terminal: Play audio data.

[1674] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1675] Example 2: Checking the weather forecast

[1676] 1. User: "What's the weather like today?"

[1677] 2. Device: Uses microphone to capture audio and send it to the server.

[1678] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[1679] 4. Server: The natural language processing engine queries the weather API and generates a response: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[1680] 5. Server: Convert the text response into audio data using a speech synthesis API.

[1681] 6. Server: Sends audio data to the device.

[1682] 7. Terminal: Play audio data.

[1683] 8. User: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[1684] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1685] Step 1:

[1686] The user asks questions or gives instructions by voice.

[1687] Input: User utterance "What's the weather like today?"

[1688] Specific actions: The user speaks through Bluetooth earphones or audio glasses.

[1689] Step 2:

[1690] The terminal captures the user's speech.

[1691] Input: User's spoken utterance

[1692] What it does: It uses a built-in microphone to capture the user's speech as digital audio data.

[1693] Output: Digital audio data

[1694] Step 3:

[1695] The terminal transmits the captured audio data to the server.

[1696] Input: Digital audio data

[1697] What it does: It compresses digital audio data and sends it over the network to a server.

[1698] Output: Compressed audio data

[1699] Step 4:

[1700] The server passes the received voice data to the voice recognition API.

[1701] Input: Compressed audio data

[1702] Specific operation: Decodes the audio data and passes it to a speech recognition API such as the Google Speech-to-Text API.

[1703] Output: Text data "What's the weather like today?"

[1704] Step 5:

[1705] The server inputs the data converted into text by the speech recognition API into a natural language processing engine.

[1706] Input: Text data "What's the weather like today?"

[1707] What it does: Provides text data as prompts to a natural language processing engine such as OpenAI GPT-3.

[1708] Output: Response "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1709] Step 6:

[1710] The server passes the generated response text to the speech synthesis API.

[1711] Input: Response "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees."

[1712] What it does: Passes the response text to a speech synthesis API, such as the Google Text-to-Speech API, to generate natural-sounding speech.

[1713] Output: Audio data

[1714] Step 7:

[1715] The server transmits the generated voice data to the user's terminal.

[1716] Input: Audio data

[1717] Specific operations: Encodes audio data and sends it over the network to the user's device.

[1718] Output: Audio data

[1719] Step 8:

[1720] The terminal decodes the received audio data and plays it on the speaker.

[1721] Input: Received audio data

[1722] Specific behavior: Decodes audio data and plays it to the user through the built-in speaker.

[1723] Output: Audio played: "Today it will be cloudy with occasional rain, with a high of 22 degrees."

[1724] (Application example 1)

[1725] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1726] Current voice interfaces for autonomous vehicles have difficulty accurately analyzing user voice input and providing appropriate information. Furthermore, there is a lack of technology that allows drivers to efficiently operate autonomous vehicles through voice commands, making it difficult to balance safe operation with information provision. Therefore, there is a demand for a system that can improve the accuracy of voice interfaces within autonomous vehicles and provide appropriate information needed during operation.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1728] In this invention, the server includes means for capturing voice input from a user, means for transmitting the voice input to the server, and means for converting the voice input into text data, thereby enabling generation of appropriate route guidance based on position information within the autonomous mobile body.

[1729] "Voice input" is voice data uttered by the user through a microphone.

[1730] A "server" is a computer system that analyzes voice and text data over a network and generates an appropriate response.

[1731] "Text data" is character information converted by speech recognition.

[1732] A "response" is information or instructions generated based on voice input.

[1733] "Audio data" is data in an audio format that is converted from text data by voice synthesis.

[1734] An "autonomous mobile object" is a machine such as a vehicle or robot that moves autonomously without human operation.

[1735] "Route guidance" refers to instructions that include route information to a destination.

[1736] "Means for analyzing in natural language" refers to methods and technologies for analyzing and understanding voice input using language processing technology.

[1737] A "map information processing engine" is a system that analyzes map data and provides route guidance and identifies the current location.

[1738] Overall system configuration

[1739] This system consists of a voice interface device installed in the autonomous vehicle and an external cloud server. Users can communicate with the autonomous vehicle through the voice interface and receive various information and instructions.

[1740] Hardware Configuration

[1741] Voice interface device for autonomous mobile body

[1742] The audio interface device includes the following hardware:

[1743] Microphone: Captures audio input.

[1744] Speaker: Plays back the generated audio data.

[1745] Communication module: Sends and receives data to and from the server.

[1746] server

[1747] It exists on the cloud and has the following functions:

[1748] Speech Recognition API: Converts voice data into text.

[1749] Natural language processing engine: Analyzes text and generates responses.

[1750] Map information processing engine: Generates route guidance based on location information.

[1751] Speech synthesis API: Converts text into audio data.

[1752] Software configuration and data processing

[1753] 1. Capture voice input

[1754] When a user speaks into the voice interface device, a built-in microphone captures the voice and transmits the data to a server.

[1755] 2. Converting voice data to text

[1756] The server converts the transmitted voice data into text data using a speech recognition API, which performs spectral analysis of the voice signal and extracts the spoken content as a string of characters.

[1757] 3. Natural Language Processing and Response Generation

[1758] The text data is fed into a natural language processing engine, which analyzes the question or instruction and generates an appropriate response. For example, if the user requests route guidance, a map information processing engine is called to calculate the optimal route.

[1759] 4. Generate response audio data

[1760] The generated response text is passed to a speech synthesis API and converted into audio data, which takes into account the grammatical structure and intonation of the text to generate natural-sounding speech.

[1761] 5. Sending and Playing Audio Data

[1762] The generated voice data is transmitted to the voice interface device through the communication module of the autonomous mobile body and played back through a speaker.

[1763] Specific examples

[1764] 1. Providing route guidance

[1765] User: "What's the quickest route to my house?"

[1766] Server: Converts speech to text using speech recognition API.

[1767] Server: The natural language processing engine analyzes the request "Tell me the shortest route to my house" and calls the map information processing engine to obtain route information.

[1768] Server: Convert the generated route guidance into voice data using a speech synthesis API.

[1769] Autonomous mobile device: Receives and plays back audio data.

[1770] 2. Examples of prompts

[1771] "Please tell me the shortest route to my destination. Please specify the fastest route. Starting point: current location, destination: home. Please also take current traffic information into consideration."

[1772] This system allows users to efficiently obtain information by voice without operating the vehicle, improving safety while driving.

[1773] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1774] Step 1:

[1775] Capture voice input

[1776] Input: User utterance

[1777] Output: Audio data

[1778] Specific operation: When a user speaks to the voice interface device, the terminal (interface device in the autonomous mobile body) captures the voice using a microphone. The captured voice is sent as digital data to the server via the communication module.

[1779] Step 2:

[1780] Converting audio data to text

[1781] Input: Audio data

[1782] Output: Text data

[1783] Specific operation: The server inputs the received voice data into the voice recognition API. The voice recognition API performs spectral analysis of the voice signal and generates corresponding text data. For example, if a user says, "Tell me the shortest route to my house," the voice data is converted into text data saying, "Tell me the shortest route to my house."

[1784] Step 3:

[1785] Natural Language Processing and Response Generation

[1786] Input: Text data

[1787] Output: Response text

[1788] Specific operation: The server inputs the converted text data into a natural language processing engine. The natural language processing engine analyzes the text data and understands the user's instructions and questions. For example, in response to an input such as "What is the shortest route to my house?", the engine calls a map information processing engine and generates appropriate route guidance. The generated response text is "The shortest route from my current location to my house is..."

[1789] Step 4:

[1790] Generate response audio data

[1791] Input: Response text

[1792] Output: Audio data

[1793] Specific operation: The server passes the generated response text to the speech synthesis API. The speech synthesis API takes into account the grammatical structure and intonation of the text and generates natural-sounding speech data. For example, a text response such as "The shortest route from your current location to your home is..." is converted into speech data.

[1794] Step 5:

[1795] Sending and playing audio data

[1796] Input: Audio data

[1797] Output: Audio output to the user

[1798] Specific operation: The generated voice data is sent from the server to the voice interface device via the communication module of the autonomous mobile device. The terminal plays the received voice data through a speaker and provides a response to the user. For example, the user can hear a voice response such as, "The shortest route from your current location to your home is..."

[1799] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1800] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[1801] Overall system configuration

[1802] This system consists of a user device (such as Bluetooth earphones or audio glasses), a server located in the cloud, and an emotion engine. The user asks questions or gives instructions via voice through the device, and the voice data is sent to the server via the network. The server converts the voice data into text, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[1803] Detailed program processing

[1804] Server Processing

[1805] 1. Receiving voice input

[1806] The server receives the voice data sent from the device, which is first decoded and converted into an analyzable format.

[1807] 2. Converting voice data to text

[1808] The server converts the voice data into text data using a speech recognition API, performs spectral analysis of the voice signal, and extracts the spoken content as a string of characters.

[1809] 3. Emotion Analysis

[1810] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[1811] 4. Natural Language Processing and Response Generation

[1812] The server inputs the text data into a natural language processing engine to analyze the user's intent. It also takes into account the results of sentiment analysis and adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[1813] 5. Generating Audio Data

[1814] The server then passes the generated text response to a speech synthesis API, which converts it into audio data, adjusting intonation and tone based on the results of sentiment analysis.

[1815] 6. Sending audio data

[1816] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[1817] Terminal handling

[1818] 1. Capture voice input

[1819] The device detects the user's speech and captures audio input using a built-in microphone, which is then sent as digital data to a server.

[1820] 2. Receiving and playing audio data

[1821] The terminal receives the voice data sent from the server, and the voice data is played back through the terminal's speaker, allowing the user to hear the response from the server through the played back voice.

[1822] Specific examples

[1823] Example 1: Checking the schedule

[1824] 1. User: "What's my schedule for tomorrow?"

[1825] 2. Device: Uses microphone to capture audio and send it to the server.

[1826] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's tomorrow's schedule?"

[1827] 4. Server: The emotion engine recognizes that the user is nervous.

[1828] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10:00 AM." This is converted into voice data using the speech synthesis API in a gentle tone.

[1829] 6. Server: Sends audio data to the device.

[1830] 7. Terminal: Play audio data.

[1831] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1832] Example 2: Checking the weather forecast

[1833] 1. User: "What's the weather like today?"

[1834] 2. Device: Uses microphone to capture audio and send it to the server.

[1835] 3. Server: Convert the speech to text using a speech recognition API. Say, "What's the weather like today?"

[1836] 4. Server: The emotion engine recognizes that the user is calm.

[1837] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[1838] 6. Server: Sends audio data to the device.

[1839] 7. Terminal: Play audio data.

[1840] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1841] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[1842] The processing flow will be explained below.

[1843] This invention is a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile lines, and receive responses by voice. By combining it with an emotion engine, it provides the ability to recognize the user's emotions and generate more appropriate responses.

[1844] Detailed program processing

[1845] Step 1:

[1846] The user speaks into the Bluetooth earphones or audio glasses to ask questions or give instructions, such as "Tell me the weather today."

[1847] Step 2:

[1848] The device uses a built-in microphone to capture the user's voice input, which is then converted into a digital format and transmitted over a network to a server.

[1849] Step 3:

[1850] The server receives the audio data sent from the terminal, which is first decoded and converted into an analyzable format.

[1851] Step 4:

[1852] The server uses a speech recognition API (e.g., a speech recognition engine) to convert the voice data into text data. Specifically, it performs spectral analysis of the voice signal, recognizes phonemes, and then extracts the spoken content as a string of characters. In this example, the text data generated is "Tell me what the weather is today."

[1853] Step 5:

[1854] The server uses an emotion engine to analyze the tone, speed, intonation, etc. of the voice data to recognize the user's emotion, for example, whether the user is excited, sad, calm, etc.

[1855] Step 6:

[1856] The server inputs the text data into a natural language processing engine (e.g., a natural language processing platform) to analyze the user's intent. Taking into account the results of sentiment analysis, the server adjusts the content and tone of the response. For example, if the user is tired, it generates a gentler response.

[1857] Step 7:

[1858] The server sends a query to the weather forecast API to obtain current weather information. The response from the API includes information such as the weather and temperature. Based on this response, it generates a text response such as "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1859] Step 8:

[1860] The server passes the generated text response to a speech synthesis API (e.g., a speech synthesis engine) and converts it into audio data. The speech synthesis API generates natural-sounding speech by taking into account the grammatical structure and intonation of the text. It also adjusts the intonation and tone based on the results of sentiment analysis.

[1861] Step 9:

[1862] The server re-encodes the generated voice data and converts it into a format that can be transmitted to the terminal, and transmits the converted voice data to the terminal via the network.

[1863] Step 10:

[1864] The terminal receives the audio data sent from the server, decodes the received audio data, and converts it into a playable format.

[1865] Step 11:

[1866] The device plays the audio data using a built-in speaker (e.g., a speaker in earphones or audio glasses), allowing the user to hear the response, "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees."

[1867] Specific examples

[1868] Example 1: Checking the schedule

[1869] 1. User: "What's my schedule for tomorrow?"

[1870] 2. Device: Uses microphone to capture audio and send it to the server.

[1871] 3. Server: Use a speech recognition API to convert the speech into text, saying "What's my schedule for tomorrow?"

[1872] 4. Server: The emotion engine recognizes that the user is nervous.

[1873] 5. Server: The natural language processing engine calls the schedule management API and generates a response, such as "I have a meeting tomorrow at 10 AM." The response is converted into voice data using a speech synthesis API in a gentle tone.

[1874] 6. Server: Sends the audio data to the device.

[1875] 7. Terminal: Plays audio data.

[1876] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1877] Example 2: Checking the weather forecast

[1878] 1. User: "What's the weather like today?"

[1879] 2. Device: Uses microphone to capture audio and send it to the server.

[1880] 3. Server: Use the speech recognition API to convert the speech into text, saying "What's the weather like today?"

[1881] 4. Server: The emotion engine recognizes that the user is calm.

[1882] 5. Server: The natural language processing engine calls the weather forecast API and generates a response, such as "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API in a normal tone.

[1883] 6. Server: Sends the audio data to the device.

[1884] 7. Terminal: Plays audio data.

[1885] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1886] This system allows users to efficiently obtain information using only their voice, without having to manually operate a device, and to receive responses that correspond to their emotional state at the time. This not only allows users to continue working without interrupting other tasks, but also provides great convenience by enabling responses that take the user's emotions into consideration.

[1887] Example 2

[1888] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1889] While modern voice assistant systems typically provide responses based on user utterances, they rarely consider the user's emotional state. As a result, appropriate responses are not provided based on the user's emotional state, resulting in a poor user experience. In addition, the system's inability to recognize the user's emotions makes it difficult to provide more human-like interactions.

[1890] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1891] In this invention, the server includes a means for converting voice input into text data, a means for recognizing the user's emotions based on the text data, and a means for analyzing the user's intentions and generating an appropriate response according to the emotions, thereby enabling a more natural and human-like response that takes the user's emotions into consideration.

[1892] A "user" is a person who uses the system to provide voice input and receive responses.

[1893] "Voice input" refers to a voice signal of a question or instruction verbally given by the user.

[1894] A "capturing means" is a device or method for converting a user's voice input into digital data and acquiring it.

[1895] A "server" is a central object that works in conjunction with terminals via a network to analyze voice data, convert it into text, analyze emotions, and generate responses.

[1896] "Text data" is voice data converted into a character string, and is generated by voice recognition.

[1897] A "converting means" is a technique or device used to convert voice data into text data or text data into voice data.

[1898] "Means for recognizing emotions" refers to technologies and algorithms that analyze text data and voice data to determine a user's emotional state.

[1899] "Means for analyzing intent" refers to techniques and methods for understanding what a user wants based on the content of their speech and their emotional state, and generating an appropriate response.

[1900] "Means for generating a response" refers to techniques and methods for generating appropriate information or answers in response to the analyzed user's intentions and emotions.

[1901] "Means for reproducing audio data" refers to a device or method for providing the generated audio data to the user in a reproducible format.

[1902] The present invention aims to develop a system that allows users to ask questions and give instructions to AI by voice via Bluetooth earphones or audio glasses with built-in mobile connections, and receive responses by voice. By further combining this with an emotion engine, the system can recognize the user's emotions and generate more appropriate responses.

[1903] This system consists of a user's device and a server and emotion engine located on the cloud. The user speaks questions or commands through the device, and the voice data is sent to the server via the network. The server converts the voice data into text data, analyzes it using a natural language processing engine, and generates an appropriate response. The generated text response is then converted back into voice data and sent to the user's device. The device then plays back the received voice data for the user to hear. The emotion engine is then used to analyze the user's emotional state and adjust the content and tone of the response based on that.

[1904] Hardware and software used

[1905] 1. Terminal

[1906] Bluetooth earphones and audio glasses are used to capture the user's voice. These devices use a built-in microphone to convert audio input into digital data, which is then sent to a server over the internet.

[1907] 2. Server

[1908] Speech recognition APIs: Used to convert voice data into text data. Examples include Google Cloud Speech-to-Text and Azure Speech Service.

[1909] Natural language processing engines: Used to analyze text data and identify user intent. Examples include OpenAI GPT-3 and Amazon Lex.

[1910] Emotion engines: Used to analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Affectiva.

[1911] Text-to-speech APIs: Used to convert text data into speech data, such as Amazon Polly and Google Cloud Text-to-Speech.

[1912] Specific examples

[1913] Example 1: Checking the schedule

[1914] 1. User: "What's my schedule for tomorrow?"

[1915] 2. Device: Uses microphone to capture audio and send it to the server.

[1916] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's tomorrow's schedule?"

[1917] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is nervous.

[1918] 5. Server: Generate a response using a natural language processing engine (OpenAI GPT-3) saying, "We have a meeting tomorrow at 10:00 AM." Convert this into voice data using a speech synthesis API (Amazon Polly) in a gentle tone.

[1919] 6. Server: Sends audio data to the device.

[1920] 7. Terminal: Play audio data.

[1921] 8. User: Hears, "I have a meeting tomorrow at 10 AM."

[1922] Example 2: Checking the weather forecast

[1923] 1. User: "What's the weather like today?"

[1924] 2. Device: Uses microphone to capture audio and send it to the server.

[1925] 3. Server: Convert the speech to text using a speech recognition API (Google Cloud Speech-to-Text). Say, "What's the weather like today?"

[1926] 4. Server: The emotion engine (IBM Watson Tone Analyzer) recognizes that the user is calm.

[1927] 5. Server: A natural language processing engine (OpenAI GPT-3) generates a response, stating, "Today it will be cloudy with occasional rain, with a maximum temperature of 22 degrees." This is converted into voice data using a speech synthesis API (Amazon Polly) in a normal tone.

[1928] 6. Server: Sends audio data to the device.

[1929] 7. Terminal: Play audio data.

[1930] 8. User: "Today it will be cloudy with occasional rain and a high of 22 degrees."

[1931] This system allows users to obtain information efficiently and naturally without manually operating the device, and to receive responses that correspond to their emotional state at the time. This provides great convenience as it does not require users to interrupt other tasks and can respond in a way that takes into account the user's emotions.

[1932] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1933] Step 1:

[1934] Capture audio input

[1935] The user puts on Bluetooth earphones or audio glasses and asks questions or gives instructions such as "What's the weather like today?"

[1936] Input: User speech.

[1937] The device uses a built-in microphone to capture the user's speech as audio data, which is then converted into a digital format.

[1938] Output: Audio data in digital format.

[1939] Step 2:

[1940] Sending audio data

[1941] Input: Audio data in digital format.

[1942] The device transmits the captured audio data to a server over a secure network connection.

[1943] Output: The audio data sent to the server.

[1944] Step 3:

[1945] Receiving and decoding audio data

[1946] Input: Audio data sent from the device.

[1947] The server receives the audio data sent from the device. The received audio data is decoded into an analyzable format. This decoding process makes the audio data easier to handle as a digital signal.

[1948] Output: Decoded audio data in a parsable format.

[1949] Step 4:

[1950] Converting audio data to text

[1951] Input: Decoded audio data.

[1952] The server calls a speech recognition API (such as Google Cloud Speech-to-Text) to convert the audio data into text data. During this process, the audio signal is subjected to spectral analysis and the spoken content is extracted as a string of characters.

[1953] Output: Text data (e.g. "What's the weather like today?").

[1954] Step 5:

[1955] Emotion Analysis

[1956] Input: Text and audio data.

[1957] The server uses an emotion engine (such as IBM Watson Tone Analyzer) to analyze the tone, rate, and intonation of the voice data, thereby recognizing the user's emotional state, such as whether they are nervous, excited, or calm.

[1958] Output: The user's emotional state (e.g., calm).

[1959] Step 6:

[1960] Natural Language Processing and Response Generation

[1961] Input: Text data and sentiment analysis results.

[1962] The server inputs the text data into a natural language processing engine (such as OpenAI GPT-3) to analyze the user's intent. Taking into account the results of sentiment analysis, the server generates an appropriate response and tone. For example, if the user asks, "What's the weather like today?", the response will be, "It's cloudy with occasional rain today, with a maximum temperature of 22 degrees," and if the user is calm, the response should be in a normal tone.

[1963] Output: Response text (e.g., "Today it will be cloudy with occasional rain, with a high of 22 degrees.").

[1964] Step 7:

[1965] Generating audio data

[1966] Input: Response text and sentiment analysis results.

[1967] The server then passes the generated text response to a speech synthesis API (such as Amazon Polly) and converts it into speech data. During speech synthesis, the server also uses the results of emotion analysis to set appropriate intonation and tone.

[1968] Output: Response audio data.

[1969] Step 8:

[1970] Encoding and transmitting audio data

[1971] Input: Response audio data.

[1972] The server encodes the generated voice data for transmission to the terminal, and the encoded voice data is sent back to the terminal via the network.

[1973] Output: The audio data sent to the device.

[1974] Step 9:

[1975] Receiving and playing audio data

[1976] Input: Audio data sent from the server.

[1977] The device receives the audio data sent from the server, decodes it into a playable format, and plays the decoded audio through the device's speaker.

[1978] Output: The audio data to be played (e.g., "Today it will be cloudy with occasional rain, and the maximum temperature will be 22 degrees.").

[1979] (Application example 2)

[1980] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1981] Modern voice interfaces are capable of generating appropriate responses to user inputs. However, these systems are unable to consider the user's emotional state, and the responses are often inappropriate for the user's situation. This results in a poor user experience, as the system is unable to provide intuitively desirable responses based on the user's emotional state. Furthermore, it is difficult to provide personalized suggestions based on the user's emotional state, especially in services such as food delivery.

[1982] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1983] In this invention, the server includes means for capturing voice input, means for converting the voice input from voice to text data, means for recognizing the emotional state of the user, means for generating an appropriate response from the text data taking the emotional state into consideration, means for converting the generated response from text to voice data, and means for transmitting the voice data to the user, thereby enabling personalized content suggestions based on the user's emotional state.

[1984] "Means for capturing audio input" refers to a microphone or sensor that detects the audio emitted by the user and converts it into digital data.

[1985] The "means for transmitting audio input to a server" is a component that has the function of transmitting captured audio data to a remote server via a communication network such as the Internet.

[1986] "Means for converting voice input into text data" refers to software or hardware that uses voice recognition technology to convert received voice data into text.

[1987] "Means for generating an appropriate response" refers to an algorithm or program that analyzes the user's input intent and generates answers or suggestions to the user's questions or requests based on the analysis results.

[1988] The "means for converting a response into voice data" is a system for converting the generated text-format response into voice data using a natural language processing engine or voice synthesis technology.

[1989] The "means for transmitting voice data to a user" refers to a component having a function for transmitting voice data to a user's terminal via a network.

[1990] "Means for reproducing audio data" refers to a speaker or earphone for reproducing audio data received at a user terminal.

[1991] The "means for recognizing emotional state" is a system that includes machine learning models and algorithms for analyzing tone, intonation, facial expressions, etc. from a user's voice data to identify the user's emotional state.

[1992] "Means for generating responses based on emotional state" refers to a program or algorithm for generating responses that take into account the user's perceived emotional state and have a tone and content appropriate to the user's situation.

[1993] A "means for suggesting personalized content based on a user's emotional state" is a system or algorithm for providing appropriate, individualized content or suggestions based on a user's emotional state.

[1994] The present invention is a voice dialogue system including user emotion recognition, particularly adapted for food delivery applications.

[1995] Overall system configuration

[1996] The system mainly consists of the following components:

[1997] 1. A device that captures the user's voice

[1998] 2. Means of transmitting audio data from the device to the server

[1999] 3. Server-side speech recognition, natural language processing, and emotion recognition

[2000] 4. Emotion-based response generation

[2001] 5. A means of converting the response into audio data and sending it to the device

[2002] 6. How to play audio data on the device

[2003] Program processing explanation

[2004] User side (terminal)

[2005] A user inputs voice using a device such as a smartphone or smart glasses. The device captures the voice using a built-in microphone and transmits the voice data to a server via a network.

[2006] Server side

[2007] The server does the following:

[2008] 1. Speech recognition: Use a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert voice data into text data.

[2009] 2. Emotion Recognition: An emotion engine (e.g., cognitive emotion recognition model) analyzes the user's emotional state (e.g., joy, sadness, anger, etc.) from text and voice data.

[2010] 3. Natural Language Processing: A natural language processing engine (e.g., OpenAI GPT model) is used to understand the user's intent and generate an appropriate response based on that. The generated response is adjusted in content and tone based on the user's emotional state.

[2011] 4. Convert the response to speech: Use a text-to-speech API (e.g., Amazon Polly) to convert the text response to audio data.

[2012] 5. Sending voice data: The generated voice data is sent again to the user's terminal via the network.

[2013] Actual system usage example

[2014] For example, the system works in the following example:

[2015] What to do if you're tired

[2016] 1. User: "I feel kind of tired..."

[2017] 2. Device: Uses microphone to capture audio and send it to the server.

[2018] 3. Server: Converts the speech into text using a speech recognition API. The server recognizes the phrase "I feel tired."

[2019] 4. Server: The emotion engine recognizes that the user is tired.

[2020] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as, "Cheer up, how about a nutritious salad bowl?"

[2021] 6. Server: Convert into voice data using speech synthesis API.

[2022] 7. Server: Sends audio data to the device.

[2023] 8. Device: Play audio data. The audio says, "Cheer up! How about a nutritious salad bowl?"

[2024] What to do if things are calm

[2025] 1. User: "What should I have for lunch?"

[2026] 2. Device: Uses microphone to capture audio and send it to the server.

[2027] 3. Server: Converts speech to text using a speech recognition API. Recognizes "What should I have for lunch?"

[2028] 4. Server: The emotion engine recognizes that the user is calm.

[2029] 5. Server: The natural language processing engine takes into account the emotional state and generates a response such as "You look great today! How about some salmon sushi?"

[2030] 6. Server: Convert into voice data using speech synthesis API.

[2031] 7. Server: Sends audio data to the device.

[2032] 8. Device: Play the audio data. The audio will say, "You look great today! How about some salmon sushi?"

[2033] Prompt Sentence Examples

[2034] Example prompt 1: "Generate a dialogue that suggests a menu to help the user relax when they are tired."

[2035] Example prompt 2: "Generate a dialogue that suggests an energizing meal for the user when they are calm."

[2036] In this way, a system can be implemented that provides personalized responses that take into account the user's emotions.

[2037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2038] Step 1:

[2039] The user performs voice input. The user issues instructions or asks questions by voice to a device such as a smartphone or smart glasses. The input data is the user's voice (voice signal).

[2040] Step 2:

[2041] The device captures audio input by using the device's built-in microphone to convert the user's voice into digital data, which is raw audio data (such as WAV or MP3 format).

[2042] Step 3:

[2043] The device sends the captured audio data to the server. The digital audio data is sent to the server via a communication network (Wi-Fi or mobile data communication). The input is digital audio data, and the output is the same data transferred to the server.

[2044] Step 4:

[2045] The server converts the received voice data into text using a speech recognition API, such as Google Cloud Speech-to-Text, which analyzes the voice signal and generates a corresponding string of characters. The input is digital voice data, and the output is text instructions or questions.

[2046] Step 5:

[2047] The server inputs the text data into the emotion engine to recognize the user's emotional state. The emotion engine uses a machine learning model to analyze the user's emotions (happiness, sadness, anger, etc.) from the text. The input is text data, and the output is metadata that indicates the user's emotional state.

[2048] Step 6:

[2049] The server uses a natural language processing engine to analyze the user's intent and generate an appropriate response based on that. It also takes into account the results of sentiment analysis to adjust the content and tone of the response. For example, it uses the OpenAI GPT model to generate answers and suggestions to the user's questions. The input is text data and sentiment metadata, and the output is a sentiment-adjusted text response.

[2050] Step 7:

[2051] The server converts the generated text response into audio data using a speech synthesis API. Services such as Amazon Polly are used to synthesize the text into natural-sounding speech. The input is the text response, and the output is audio data (in a playable format).

[2052] Step 8:

[2053] The server encodes the voice data and sends it to the terminal. The generated voice data is then sent back to the user's terminal via the network. The input is the encoded voice data, and the output is the same data transferred to the terminal.

[2054] Step 9:

[2055] The device plays the received audio data. The audio data is played to the user using the device's speaker or earphones. The input is digital audio data, and the output is played back in a form that the user can hear as sound.

[2056] These steps allow users to operate a food delivery application using only their voice and receive personalized suggestions based on their emotions.

[2057] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2058] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2059] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2060] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2061] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2062] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2063] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2064] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2065] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2066] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2067] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2068] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2069] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2070] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2071] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2072] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2073] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2074] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2075] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2076] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2077] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2078] The following is further disclosed regarding the above embodiment.

[2079] (Claim 1)

[2080] means for capturing voice input from a user;

[2081] means for transmitting the speech input to a server;

[2082] means for converting the speech input into text data;

[2083] means for generating an appropriate response based on the text data;

[2084] means for converting the response into audio data;

[2085] means for transmitting the audio data to a user;

[2086] means for reproducing the audio data;

[2087] A system including:

[2088] (Claim 2)

[2089] 10. The system of claim 1, further comprising means for analyzing the voice input in natural language.

[2090] (Claim 3)

[2091] 10. The system of claim 1, wherein the means for generating a response uses an external natural language processing engine.

[2092] "Example 1"

[2093] (Claim 1)

[2094] means for obtaining speech input from a user;

[2095] means for transmitting the speech input to a processing unit;

[2096] means for converting the voice input into text data;

[2097] means for generating an appropriate response based on the character data;

[2098] means for converting the response into audio data;

[2099] means for transmitting the audio data to a user;

[2100] means for reproducing the audio data;

[2101] A system including:

[2102] (Claim 2)

[2103] 10. The system of claim 1, further comprising means for analyzing the voice input in natural language.

[2104] (Claim 3)

[2105] 10. The system of claim 1, wherein the means for generating a response uses an external language processing engine.

[2106] "Application Example 1"

[2107] (Claim 1)

[2108] means for capturing voice input from a user;

[2109] means for transmitting the speech input to a server;

[2110] means for converting the speech input into text data;

[2111] means for generating an appropriate response based on the text data;

[2112] means for converting the response into audio data;

[2113] means for transmitting the audio data to a user;

[2114] means for reproducing the audio data;

[2115] a means for generating appropriate route guidance based on position information within the autonomous moving body;

[2116] A system including:

[2117] (Claim 2)

[2118] 10. The system of claim 1, further comprising means for analyzing the voice input in natural language.

[2119] (Claim 3)

[2120] 2. The system of claim 1, wherein the means for generating a response uses an external natural language processing engine and a map information processing engine.

[2121] "Example 2: Combining Emotion Engines"

[2122] (Claim 1)

[2123] means for capturing voice input from a user;

[2124] means for transmitting the speech input to a server;

[2125] means for converting voice input into text data;

[2126] A means for recognizing user emotions based on text data;

[2127] A means for analyzing a user's intention and generating an appropriate response according to the user's emotion;

[2128] means for converting the response into audio data;

[2129] means for transmitting the audio data to a user;

[2130] means for reproducing the audio data;

[2131] A system including:

[2132] (Claim 2)

[2133] 10. The system of claim 1, further comprising means for analyzing the speech input in natural language and adjusting the response taking into account the results of the sentiment analysis.

[2134] (Claim 3)

[2135] 10. The system of claim 1, wherein the means for generating a response uses an external natural language processing engine.

[2136] "Application example 2 when combining emotion engines"

[2137] (Claim 1)

[2138] means for capturing voice input from a user;

[2139] means for transmitting the speech input to a server;

[2140] means for converting the speech input into text data;

[2141] means for generating an appropriate response based on the text data;

[2142] means for converting the response into audio data;

[2143] means for transmitting the audio data to a user;

[2144] means for reproducing the audio data;

[2145] means for recognizing an emotional state and generating a response based on the emotional state;

[2146] means for suggesting personalized content according to the emotional state of the user;

[2147] A system including:

[2148] (Claim 2)

[2149] 10. The system of claim 1, further comprising means for analyzing the voice input in natural language.

[2150] (Claim 3)

[2151] 10. The system of claim 1, wherein the means for generating a response uses an external natural language processing engine. [Explanation of symbols]

[2152] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing voice input from a user; means for transmitting the speech input to a server; means for converting the speech input into text data; means for generating an appropriate response based on the text data; means for converting the response into audio data; means for transmitting the audio data to a user; means for reproducing the audio data; A system including:

2. 10. The system of claim 1, further comprising means for analyzing the voice input in natural language.

3. 10. The system of claim 1, wherein the means for generating a response uses an external natural language processing engine.

Images