System

Abstract

A system is provided.SOLUTION: A system, comprising: means for obtaining speech input; means for converting the speech input into text data; means for parsing the text data and generating appropriate response text; means for converting the response text into speech data; and means for playing the speech data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People with hearing limitations experience significant inconveniences in communicating in their daily lives. While they often use written or sign language to understand what others are saying, these methods are only available in limited situations and often result in a disruptive flow of communication. Furthermore, typical speech translation systems are limited to one-way speech recognition and translation, making two-way communication difficult. [Means for solving the problem]

[0005] The present invention is a system including means for receiving a voice input, means for converting the voice input into text data, means for analyzing the text data and generating an appropriate response text, means for converting the response text into voice data, and means for playing the voice data. In particular, by including means for receiving voice data and converting it into text data for analysis, means using artificial intelligence to analyze the text data and generate an appropriate response, means for converting the generated response into voice data, and means for transmitting the voice data, people with hearing limitations can smoothly engage in two-way communication with others.

[0006] "Audio input" refers to the process or device that converts audio into electrical signals and the resulting data.

[0007] "Text data" refers to character string information converted from audio data, and includes character sequences based on natural language.

[0008] "Means for converting" refers to hardware or software processes and devices for converting data of one format to data of another format.

[0009] "Means for analyzing" refers to a hardware or software process that includes computational methods or algorithms for semantically understanding or processing input data.

[0010] "Means for generating" refers to the algorithm or process for generating new data based on input data and the results thereof.

[0011] "Artificial intelligence means" refers to systems and algorithms that use techniques such as machine learning and deep learning to analyze and generate data.

[0012] "Audio data" refers to data of an audio signal represented in analog or digital form.

[0013] "Playback means" refers to processes and devices for converting audio data into a form that can be heard through a speaker or other device.

[0014] "Transmission means" refers to the communications protocols, devices, and processes used to transfer data from one point to another. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system for assisting people with hearing limitations in their daily communication by converting speech input into text data, generating appropriate responses, and converting them back into speech data for presentation to the user.

[0037] System configuration

[0038] 1. Acquiring voice input (device)

[0039] The device uses a microphone to receive user voice input, which is captured as digital audio data.

[0040] For example, a user might say, "Please give me a plastic bag at the convenience store."

[0041] 2. Sending voice data (terminal → server)

[0042] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[0043] 3. Speech recognition (server)

[0044] The server processes the received voice data and converts it into text data using OpenAI's Whisper API.

[0045] For example, speech data uttered "Please give me a plastic bag at the convenience store" is converted into text data stating "Please give me a plastic bag at the convenience store."

[0046] 4. Text analysis and response generation (server)

[0047] The server then uses artificial intelligence to analyze the converted text data, which allows it to understand the meaning of the text and generate an appropriate response.

[0048] For example, in response to the text data "Please give me a plastic bag at the convenience store," the AI ​​generates the response "Yes, would you like a plastic bag?"

[0049] 5. Speech synthesis (server)

[0050] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data.

[0051] The response text "Yes, would you like a plastic bag?" is converted into speech data.

[0052] 6. Sending audio data (server → device)

[0053] The server transmits the generated voice data to the terminal, which receives it and provides a response to the user.

[0054] 7. Audio playback (device)

[0055] The device then plays the received voice data through the speaker, and the user hears the voice saying, "Yes, would you like a plastic bag?"

[0056] Specific examples

[0057] Scenario: A user places an order at a cafe

[0058] 1. Acquiring voice input (device)

[0059] The user speaks to the terminal, "I'd like a small iced coffee, please."

[0060] The device picks up this audio and captures it as digital audio data.

[0061] 2. Sending voice data (terminal → server)

[0062] The terminal transmits the voice data to the server.

[0063] 3. Speech recognition (server)

[0064] The server converts the speech data into text "I'd like a small iced coffee please."

[0065] 4. Text analysis and response generation (server)

[0066] The server uses artificial intelligence to analyze the text data and generate an appropriate response: "Thank you. Is small size okay?"

[0067] 5. Speech synthesis (server)

[0068] The server converts the response text into audio data.

[0069] 6. Sending audio data (server → device)

[0070] The server transmits the audio data to the terminal.

[0071] 7. Audio playback (device)

[0072] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[0073] In this way, the present invention supports everyday communication for people with hearing limitations through a series of processes that convert voice data into text data, generate an appropriate response, and provide it to the user again as voice data.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The device activates the microphone and receives voice input, for example, when the user says, "I'd like a coffee, please."

[0077] Step 2:

[0078] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[0079] Step 3:

[0080] The terminal uses an HTTP request to send the audio data to the server.

[0081] Step 4:

[0082] The server analyzes the received voice data and converts it into text data by calling a voice recognition API (e.g. Whisper API). For example, the voice saying "One coffee please" is converted into the text "One coffee please."

[0083] Step 5:

[0084] The server sends the text data to an artificial intelligence system that analyzes it to generate an appropriate response, such as "Hot or Ice?"

[0085] Step 6:

[0086] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into speech data. For example, the text "Is it hot or iced?" is converted into speech data.

[0087] Step 7:

[0088] The server packages the generated audio data and prepares an HTTP response to send to the device.

[0089] Step 8:

[0090] The server uses an HTTP response to send the audio data to the terminal.

[0091] Step 9:

[0092] The device decodes the received audio data and plays it through the speaker.

[0093] Step 10:

[0094] The user hears a voice response saying, "Hot or iced?"

[0095] The above is the specific program processing flow of the system, which allows users with hearing limitations to communicate smoothly.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] It is difficult for people with hearing limitations to communicate smoothly in their daily lives. Current communication support systems convert voice input into text, generate an appropriate response, and then convert it back into voice to provide to the user. However, these systems often lack the quality, immediacy, and appropriate response generation of the voice data. Processing delays and unstable communication are also problems. A system that can solve these issues and provide fast, high-quality communication support is needed.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes a preprocessing means for converting a voice input into digital voice data and generating high-quality voice data, a communication means for transmitting the high-quality voice data to an analysis server, a means for converting the high-quality voice data into text data, a means for analyzing the text data and generating an appropriate response text, a means for converting the response text into voice data, and a means for receiving and playing the voice data, thereby enabling the generation of high-quality voice data and instantaneous response generation.

[0101] "Audio input" refers to words or sounds made by a user through a microphone or other sound capturing device.

[0102] "Digital audio data" means data that has been converted from analog audio into a digital signal for further processing or communication.

[0103] The "pre-processing means" refers to a means for performing processes such as noise removal and echo cancellation when converting captured audio into high-quality digital audio data.

[0104] "Communication means" refers to the means for sending and receiving data between the terminal and the server. Typically, the HTTP protocol or a dedicated protocol is used via the Internet.

[0105] "Text data" refers to data in the form of a string of characters obtained by analyzing voice data. This data is in a format that is easy for computers to process.

[0106] "Analysis means" refers to artificial intelligence or algorithms that understand the content of text data and generate appropriate responses.

[0107] "Appropriate response text" refers to text data in response to text data from a user that is generated by an AI model, etc.

[0108] "Means for converting into voice data" refers to Text-to-Speech (TTS) technology or APIs for converting text data back into voice data.

[0109] The "means for playing" refers to a means for allowing the user to listen to the generated audio data through a speaker, earphones, or the like.

[0110] The present invention provides a system for supporting communication for users with hearing limitations. This system converts voice input into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. The following describes an embodiment of the present invention based on specific processing and examples.

[0111] Hardware and software used

[0112] 1. Terminal

[0113] Hardware: Microphone, speaker, communication module (Wi-Fi, 4G / 5G module, etc.)

[0114] Software: Audio capture app, digital signal processing software, communication protocol stack

[0115] 2. Server

[0116] Hardware: High-performance processor, memory, and large storage capacity

[0117] Software: Speech recognition APIs (e.g., Whisper API), text analysis engines, generative AI models (e.g., GPT-3), Text-to-Speech (TTS) APIs

[0118] Processing Description

[0119] 1. Acquiring voice input

[0120] The device uses a microphone to capture the user's voice input as digital audio data, which is then pre-processed with noise reduction and echo cancellation before being stored in a high-quality format and sent directly to the server.

[0121] 2. Sending audio data

[0122] The device then sends the captured high-quality audio data to the server, usually via HTTP requests, but sometimes using proprietary communication protocols.

[0123] 3. Voice Recognition

[0124] The server converts the received voice data into text data using OpenAI's Whisper API, which analyzes the voice data and converts it into a string format, allowing it to accurately transcribe what the user is saying.

[0125] 4. Text Analysis and Response Generation

[0126] The server analyzes the converted text data and generates an appropriate response using a generative AI model (e.g., GPT-3), which understands the user's intent and creates a response that is context-sensitive.

[0127] 5. Speech Synthesis

[0128] The server then sends the generated response text to a Text-to-Speech (TTS) API, which converts the response into audio data, possibly including selecting a voice narrator and setting emotional expressions.

[0129] 6. Sending audio data

[0130] The server then sends the generated audio data back to the device, usually via an HTTP request.

[0131] 7. Audio playback

[0132] The terminal plays the received voice data through the speaker, allowing the user to hear the response from the system.

[0133] Specific examples

[0134] Scenario: A user places an order at a cafe

[0135] Acquiring voice input

[0136] The user speaks to the device, saying, "I'd like a small iced coffee, please." The device captures this speech as high-quality digital audio data.

[0137] Sending audio data

[0138] The device transmits the captured audio data to the server.

[0139] Voice Recognition

[0140] The server uses the Whisper API to convert the received voice data into text, "I'd like a small iced coffee, please."

[0141] Text analysis and response generation

[0142] The server sends a prompt to the generative AI model to generate the appropriate response: "Thank you. Would you like the small size?"

[0143] Speech synthesis

[0144] The server converts this response text into voice data using a TTS API.

[0145] Sending audio data

[0146] The server transmits the generated voice data to the terminal.

[0147] Playing audio

[0148] The terminal plays the received audio data, and the user can hear, "Thank you. Is small size OK?"

[0149] Prompt Sentence Examples

[0150] prompt:

[0151] "If the user says, 'I'd like a small iced coffee, please,' generate an appropriate response."

[0152] As described above, the present invention makes it possible to maintain the quality of digital voice data and generate instantaneous responses, thereby smoothly supporting everyday communication for users with hearing limitations.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The device captures the user's voice input as digital voice data. For example, when the user says, "I'd like a small iced coffee, please," the microphone picks up this voice and converts the analog signal into a digital signal. The input is the user's voice, and the output is digital voice data.

[0156] Step 2:

[0157] The device preprocesses the captured digital audio data. This preprocessing includes noise reduction, echo cancellation, and audio clipping. This results in high-quality audio data. The input is digital audio data, and the output is preprocessed, high-quality digital audio data.

[0158] Step 3:

[0159] The terminal sends the preprocessed audio data to the server. The communication protocol is typically an HTTP POST request. The input is high-quality digital audio data, and the output is a response to the server confirming successful data transmission.

[0160] Step 4:

[0161] The server converts the received voice data into text data using OpenAI's Whisper API. This API analyzes the voice data and converts the user's speech into a string format. The input is high-quality digital voice data, and the output is text data. For example, "I'd like a small iced coffee, please." is converted into text.

[0162] Step 5:

[0163] The server analyzes the text data and generates an appropriate response. A generative AI model (e.g., GPT-3) is used for this analysis and response generation. The input is text data, and the output is an appropriate response text. For example, in response to "I'd like a small iced coffee, please," the server generates the response "Thank you. Is the small size okay?"

[0164] Step 6:

[0165] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts it into voice data. This API converts text data into a voice signal. The input is the response text, and the output is voice data. For example, "Thank you. Is small size okay?" is converted into voice data.

[0166] Step 7:

[0167] The server sends the generated audio data to the device. Again, an HTTP POST request is used. The input is the audio data, and the output is a response indicating successful data transmission to the device.

[0168] Step 8:

[0169] The device then plays the received audio data to the user. At this time, the audio is output using a speaker or earphones. The input is the audio data, and the output is the audio heard by the user. Specifically, the device plays back "Thank you. Is small size OK?"

[0170] (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] People with hearing limitations often face significant limitations in their daily lives, especially when communicating in brick-and-mortar stores. For example, when communicating with store clerks about orders, questions, or confirmations, they may find it difficult to hear what the other person is saying, hindering smooth communication. There is a need for technology that can solve these problems and enable people with hearing limitations to communicate smoothly in brick-and-mortar stores.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes means for acquiring a voice input, means for converting the voice input into text data, means for analyzing the text data and generating an appropriate response text, means for converting the response text into voice data, means for playing the voice data, means including an artificial intelligence model for processing the text data and the voice data, means for the system to be executed on a smart device (smartphone, smart glasses, etc.) installed in a physical store, and means for the voice data to facilitate communication in the physical store, thereby enabling people with hearing limitations to smoothly communicate in the physical store, such as when placing an order or asking a question.

[0175] "Means for acquiring voice input" refers to a device or function for capturing voice uttered by a user and inputting it into the system.

[0176] The "means for converting into text data" refers to a device or function for converting acquired voice data into text data.

[0177] The "means for generating an appropriate response text" refers to a device or function for analyzing the converted text data and generating an appropriate response based on the analyzed text data.

[0178] The "means for converting into voice data" refers to a device or function for converting the generated response text into voice data.

[0179] "Means for playing audio data" refers to the function of using a speaker or other playback device to provide the converted audio data to the user.

[0180] "Means including an artificial intelligence model" refers to a device or system function that incorporates artificial intelligence used to analyze text data and generate responses.

[0181] A "smart device" is a portable device, such as a smartphone or smart glasses, that has Internet connectivity and application execution capabilities and is used as a user interface.

[0182] "Means to be installed in a physical store" refers to the means by which a system with functions such as language and voice processing, data transmission and reception, etc. can be operated within a physical store.

[0183] "Means aimed at facilitating communication in physical stores" refers to the functions of devices and systems that improve and support everyday communication such as customer service and transactions in physical stores.

[0184] The present invention provides a system that supports people with hearing impairments to smoothly communicate in physical stores. Specific embodiments are described below in detail.

[0185] System program generation

[0186] This system supports communication through a series of processes: it receives voice input, converts it into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. Specifically, there are three main entities involved in this system: the server, the terminal, and the user.

[0187] Hardware and Software Configuration

[0188] Hardware:

[0189] Portable devices such as smartphones and smart glasses

[0190] Server (cloud-based, e.g. AWS, Google Cloud)

[0191] software:

[0192] Speech Recognition API (OpenAI's Whisper)

[0193] Text analysis and response generation model (OpenAI's GPT-4)

[0194] Speech synthesis API (Google Text-to-Speech)

[0195] Processing flow and data processing

[0196] Getting user voice input

[0197] The device captures the user's voice through the microphone. For example, the user says, "One espresso, please."

[0198] Sending audio data

[0199] The device sends the captured audio data to the server using the HTTP protocol.

[0200] Voice Recognition

[0201] The server uses OpenAI's Whisper API to convert voice data into text. For example, "One espresso please" is converted into text "One espresso please."

[0202] Text analysis and response generation

[0203] The server uses a GPT-4 model to analyze the text data and generate an appropriate response. For example, in response to "One espresso please," the server generates the response "Okay, one espresso please."

[0204] Speech synthesis

[0205] The generated response text is converted into audio data using the Google Text-to-Speech API.

[0206] Sending and playing audio data

[0207] The server sends the generated voice data to the terminal, which plays it back. The user hears the voice saying, "Okay, one espresso, please."

[0208] Specific examples

[0209] This system, which uses a combination of a server and a terminal, is applied as follows:

[0210] Case 1: Ordering at a cafe

[0211] When a user places an order at a cafe, they speak into their smartphone, saying, "One espresso, please." The device recognizes the speech and sends it to the server. The server converts it to text, generates an appropriate response, converts it to audio data, and sends it back to the device. The device then plays back the response, allowing the user to hear it and confirm that their order was received correctly.

[0212] Prompt Sentence Examples

[0213] User: One espresso please.

[0214] AI Response: Ok, one espresso please.

[0215] In this way, the system of the present invention provides a specific embodiment that enables people with hearing limitations to smoothly communicate in physical stores.

[0216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0217] Step 1:

[0218] Getting user voice input

[0219] Subject: Terminal

[0220] Specific behavior:

[0221] The device captures the user's voice through the built-in microphone.

[0222] Input: User speech (e.g., user says "One espresso please").

[0223] Data processing: The microphone converts analog audio into digital audio data.

[0224] Output: Digital audio data.

[0225] Step 2:

[0226] Sending audio data

[0227] Subject: Terminal

[0228] Specific behavior:

[0229] The device sends the captured audio data to the server.

[0230] Input: Digital audio data.

[0231] Data Calculation: Send data to the server using an HTTP request.

[0232] Output: The audio data sent to the server.

[0233] Step 3:

[0234] Voice Recognition

[0235] Subject: Server

[0236] Specific behavior:

[0237] The server converts the received voice data into text using OpenAI's Whisper API.

[0238] Input: Received audio data.

[0239] Data processing: Voice data is analyzed using the Whisper API and converted into text data.

[0240] Output: Text data (e.g., "One espresso, please").

[0241] Step 4:

[0242] Text analysis and response generation

[0243] Subject: Server

[0244] Specific behavior:

[0245] The server uses the GPT-4 model to analyze the text data and generate an appropriate response.

[0246] Input: Text data ("One espresso please").

[0247] Data computation: Semantic interpretation and response generation using the GPT-4 model.

[0248] Output: The response text data (e.g., "Okay, one espresso, please.").

[0249] Step 5:

[0250] Speech synthesis

[0251] Subject: Server

[0252] Specific behavior:

[0253] The server converts the generated response text into audio data using the Google Text-to-Speech API.

[0254] Input: Response text data.

[0255] Data processing: Convert text data into audio data using the Text-to-Speech API.

[0256] Output: Response voice data (e.g., "Okay, one espresso, please").

[0257] Step 6:

[0258] Sending audio data

[0259] Subject: Server

[0260] Specific behavior:

[0261] The server transmits the generated voice data to the terminal.

[0262] Input: Response audio data.

[0263] Data calculation: The audio data is returned to the terminal as an HTTP response.

[0264] Output: The audio data sent to the device.

[0265] Step 7:

[0266] Playing audio

[0267] Subject: Terminal

[0268] Specific behavior:

[0269] The audio data received by the terminal is played back through the speaker.

[0270] Input: Received audio data.

[0271] Data processing: The speaker converts digital audio data into an analog signal.

[0272] Output: Analogue sound signal (e.g. user hears the voice "Okay, one espresso please").

[0273] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0274] This invention is a system for enabling people with hearing limitations to communicate effectively in everyday life, and is configured with an emotion engine. This system converts voice input into text data, generates an appropriate response, and converts it back into voice data to provide to the user.

[0275] System configuration

[0276] 1. Acquiring voice input (device)

[0277] The device activates the microphone and receives the user's voice input. For example, the user might say, "Please give me a plastic bag at the convenience store." This voice input is captured as digital audio data.

[0278] 2. Sending voice data (terminal → server)

[0279] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[0280] 3. Speech recognition (server)

[0281] The server receives the voice data and converts it into text data using a speech recognition API. For example, the voice saying "Please give me a plastic bag at the convenience store" is converted into text data "Please give me a plastic bag at the convenience store."

[0282] 4. Emotion Recognition (Server)

[0283] The server inputs the text data and voice data into an emotion engine to recognize the user's emotions. For example, it recognizes the user's emotions such as "I'm in a hurry" or "I'm in a hurry" from the tone of the voice and the content of the text.

[0284] 5. Text Analysis and Response Generation (Server)

[0285] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[0286] 6. Speech synthesis (server)

[0287] The server sends the generated response text to a Text-to-Speech (TTS) API and converts it into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[0288] 7. Sending audio data (server → device)

[0289] The server packages the generated audio data and prepares an HTTP response to send to the device.

[0290] 8. Audio playback (device)

[0291] The device decodes the received voice data and plays it through the speaker, and the user can hear the voice saying "It will be ready soon."

[0292] Specific examples

[0293] Scenario: A user places an order at a cafe

[0294] 1. Acquiring voice input (device)

[0295] The user speaks to the terminal, "I'd like a small iced coffee, please."

[0296] The device picks up this audio and captures it as digital audio data.

[0297] 2. Sending voice data (terminal → server)

[0298] The terminal transmits the voice data to the server.

[0299] 3. Speech recognition (server)

[0300] The server converts the speech data into text "I'd like a small iced coffee please."

[0301] 4. Emotion Recognition (Server)

[0302] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[0303] 5. Text Analysis and Response Generation (Server)

[0304] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[0305] 6. Speech synthesis (server)

[0306] The server converts the response text into audio data.

[0307] 7. Sending audio data (server → device)

[0308] The server transmits the audio data to the terminal.

[0309] 8. Audio playback (device)

[0310] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[0311] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate corresponding appropriate responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[0312] The processing flow will be explained below.

[0313] Step 1:

[0314] The device activates the microphone and receives the user's voice input, for example, the user says, "I'd like a coffee, please."

[0315] Step 2:

[0316] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[0317] Step 3:

[0318] The terminal uses an HTTP request to send the audio data to the server.

[0319] Step 4:

[0320] The server analyzes the received voice data and converts it into text data by calling a speech recognition API (e.g., Whisper API). For example, the speech "One coffee please" is converted into the text "One coffee please."

[0321] Step 5:

[0322] The server inputs the text and voice data into an emotion engine to recognize the user's emotions. From the tone of the voice and the content of the text, the server recognizes emotions such as "I'm in a hurry" or "I'm in a hurry."

[0323] Step 6:

[0324] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[0325] Step 7:

[0326] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[0327] Step 8:

[0328] The server packages the generated audio data and prepares an HTTP response to send to the device.

[0329] Step 9:

[0330] The server uses an HTTP response to send the audio data to the terminal.

[0331] Step 10:

[0332] The device decodes the received audio data and plays it through the speaker.

[0333] Step 11:

[0334] The user hears the response, "It will be ready shortly."

[0335] Specific examples

[0336] Scenario: A user places an order at a cafe

[0337] Step 1:

[0338] The user speaks to the terminal, "I'd like a small iced coffee, please."

[0339] Step 2:

[0340] The device picks up this audio and captures it as digital audio data.

[0341] Step 3:

[0342] The terminal transmits the voice data to the server.

[0343] Step 4:

[0344] The server converts the speech data into text "I'd like a small iced coffee please."

[0345] Step 5:

[0346] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[0347] Step 6:

[0348] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[0349] Step 7:

[0350] The server converts the response text into audio data.

[0351] Step 8:

[0352] The server transmits the audio data to the terminal.

[0353] Step 9:

[0354] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[0355] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate appropriate corresponding responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[0356] Example 2

[0357] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0358] Conventional speech recognition systems can only convert speech data into text and generate responses, making it difficult to accurately grasp a user's emotions and intentions. This results in a poor user experience, particularly for people with hearing limitations. Furthermore, because they do not recognize emotions, it is difficult to provide appropriate responses based on the user's situation and feelings. As a result, communication is ineffective, often causing difficulties in daily life.

[0359] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice input into text data, a means for analyzing the text data and voice data and recognizing the user's emotion, and a means for generating an appropriate response text based on the emotion information. This makes it possible to accurately recognize the user's emotion and provide an appropriate response corresponding to that emotion.

[0360] "Audio input" refers to a voice signal emitted by a user into a device, which is captured as digital data.

[0361] "Text data" refers to text information converted from voice input using voice recognition.

[0362] "Audio data" refers to data in which an audio signal is expressed in digital form.

[0363] "Emotion recognition" refers to the process of analyzing speech and text data to identify a user's emotional state.

[0364] "Response text" refers to a textual representation of an appropriate response generated based on the user's utterance and their sentiment.

[0365] "Speech synthesis" refers to the technical process of converting text data into speech data.

[0366] "Playback" refers to audio data being provided to a user in an audible form through an output device such as a speaker.

[0367] "Transmitting and receiving" refers to the process of transferring data from one device to another.

[0368] This invention is a system that enables people with hearing limitations to communicate effectively in their daily lives, and performs a series of processes including voice data acquisition, analysis, emotion recognition, response generation, and voice playback.

[0369] Hardware and software used

[0370] Hardware:

[0371] Device: A device with a microphone and speaker, such as a smartphone or voice assistant device.

[0372] Server: Cloud environment (e.g. AWS, Google Cloud, Azure)

[0373] software:

[0374] Speech Recognition API: Google Cloud Speech-to-Text

[0375] Emotion Recognition Engine: Software that implements a specific emotion recognition algorithm (e.g., a proprietary machine learning model).

[0376] Text-to-Speech (TTS) API: Amazon Polly, etc.

[0377] System Operation Overview

[0378] 1. Acquiring voice input:

[0379] The device activates a microphone and digitally captures what the user says.

[0380] 2. Sending audio data:

[0381] The device sends the captured audio data to the server via an HTTP request.

[0382] 3. Speech Recognition:

[0383] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data.

[0384] 4. Emotion recognition:

[0385] The server inputs the text data obtained by speech recognition and the original speech data into an emotion recognition engine to recognize the user's emotions, such as "tired" or "in a hurry."

[0386] 5. Text analysis and response generation:

[0387] Based on the emotion information, the server generates an appropriate response text.

[0388] 6. Speech synthesis:

[0389] The server sends the generated response text to a Text-to-Speech API such as Amazon Polly, which converts the response into audio data.

[0390] 7. Sending audio data:

[0391] The server sends the generated audio data to the terminal as an HTTP response.

[0392] 8. Audio playback:

[0393] The terminal decodes the received audio data and plays it back to the user through the speaker.

[0394] Specific examples

[0395] Scenario: A user places an order at a cafe

[0396] For example, if a user says to their smartphone, "Please give me a small iced coffee," the system will operate as follows:

[0397] 1. Acquiring voice input:

[0398] The device captures the audio and stores it as digital audio data.

[0399] 2. Sending audio data:

[0400] The terminal transmits the voice data to the server.

[0401] 3. Speech Recognition:

[0402] The server sends the voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "I'd like a small iced coffee, please."

[0403] 4. Emotion recognition:

[0404] The server analyzes the text data and voice data and recognizes the user's emotion as "calm."

[0405] 5. Text analysis and response generation:

[0406] Based on the emotion information, the server generates a response text saying "Thank you. Is small size OK?"

[0407] 6. Speech synthesis:

[0408] The server sends this response text to Amazon Polly, which converts it into voice data.

[0409] 7. Sending audio data:

[0410] The server transmits the generated voice data to the terminal.

[0411] 8. Audio playback:

[0412] The terminal plays the audio data, and the user hears "Thank you. Is small size OK?"

[0413] Prompt Sentence Examples

[0414] Please explain a specific scenario in which a user orders a small iced coffee at a cafe, and provide a detailed explanation of each process: speech recognition, emotion recognition, response generation, speech synthesis, and speech playback.

[0415] The system enables people with hearing limitations to communicate more appropriately and effectively through emotion recognition technology.

[0416] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0417] Step 1: Getting voice input (device)

[0418] Input: User's voice

[0419] Processing: The device activates the microphone and captures the user's voice input as digital audio data.

[0420] Output: Digital audio data

[0421] Specific behavior:

[0422] The device activates the microphone and waits for the user's voice input.

[0423] A user says, "I'd like a small iced coffee at the cafe."

[0424] The device captures this audio digitally and stores it temporarily in memory.

[0425] Step 2: Sending voice data (device → server)

[0426] Input: Digital audio data

[0427] Processing: The device sends the captured audio data to the server via an HTTP POST request, using HTTPS for security reasons.

[0428] Output: Audio data sent to the server

[0429] Specific behavior:

[0430] The audio data is included in the body of the HTTPS request.

[0431] Sends a request to a specific endpoint on the server.

[0432] Step 3: Speech Recognition (Server)

[0433] Input: Audio data sent to the server

[0434] Processing: The server inputs the received audio data into the Google Cloud Speech-to-Text API and converts the audio into text data.

[0435] Output: Text data

[0436] Specific behavior:

[0437] Send the audio data to the Google Cloud Speech-to-Text API.

[0438] Receives the converted text data and passes it to the next process.

[0439] For example, the speech "I'd like a small iced coffee at the cafe, please" is converted into the text "I'd like a small iced coffee at the cafe, please."

[0440] Step 4: Emotion Recognition (Server)

[0441] Input: Text data and audio data

[0442] Processing: The server inputs the text and voice data obtained through speech recognition into an emotion recognition engine to recognize the user's emotions. For example, it may determine that the user is "calm" based on the tone of the voice and the way the words are spoken.

[0443] Output: User's emotional state

[0444] Specific behavior:

[0445] Text data and voice data are input into the emotion recognition engine.

[0446] The emotion information obtained from the emotion recognition engine is passed to the next process.

[0447] For example, the emotion "calm" is identified.

[0448] Step 5: Text analysis and response generation (server)

[0449] Input: Emotion information and text data

[0450] Processing: Based on the emotion information, the server generates an appropriate response text. For example, if the user is recognized as "calm", the response will be "Thank you. Is small size OK?"

[0451] Output: Response text

[0452] Specific behavior:

[0453] The text data and sentiment information are analyzed to determine an appropriate response text.

[0454] Generates response text and passes it to the next process.

[0455] Step 6: Speech synthesis (server)

[0456] Input: Response text

[0457] Processing: The server sends the response text to a Text-to-Speech (TTS) API (e.g., Amazon Polly) and converts it into audio data.

[0458] Output: Audio data

[0459] Specific behavior:

[0460] Send the response text to the TTS API.

[0461] The received audio data is passed to the next process.

[0462] Step 7: Sending audio data (server → device)

[0463] Input: Generated audio data

[0464] Processing: The server sends the generated audio data to the device in an HTTP response. HTTPS is used to ensure security.

[0465] Output: Audio data sent to the device

[0466] Specific behavior:

[0467] The audio data is included in the body of the HTTPS response.

[0468] Sends a response to a specific endpoint on the device.

[0469] Step 8: Playing Audio (Device)

[0470] Input: Audio data sent to the device

[0471] Processing: The device decodes the received audio data and plays it through the speaker. The user can hear the audio "Thank you. Is small size OK?"

[0472] Output: The audio the user can hear

[0473] Specific behavior:

[0474] Decodes audio data and converts it into a playable format.

[0475] Play audio data through a speaker.

[0476] The above is the specific processing flow of this system.

[0477] (Application example 2)

[0478] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0479] It is difficult for people with hearing limitations to communicate effectively in everyday life. In particular, when interacting with complex systems such as self-driving vehicles, it is necessary to generate appropriate responses that take into account the user's emotions. However, current technology has difficulty not only accepting voice input but also accurately recognizing the user's emotions and generating responses based on those emotions, resulting in a decline in the quality of communication.

[0480] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion from voice data and text data, means for generating an appropriate response text taking the emotion into consideration, and means for converting the generated response text into voice data. This makes it possible to recognize the user's emotion and generate an appropriate and prompt response based on the emotion.

[0481] A "means for obtaining voice input" is a device or method for obtaining voice from a user as digital voice data.

[0482] A "means for converting speech input into text data" is a device or method for analyzing acquired speech input and converting it into corresponding text data.

[0483] The "means for recognizing emotions" is a device or method for identifying a user's emotions from text data and voice data.

[0484] The "means for analyzing text data and generating an appropriate response text" is a device or method for analyzing text data in consideration of emotional information and generating an appropriate response.

[0485] The "means for converting response text into voice data" is a device or method for converting the generated response text into voice data.

[0486] The "means for playing audio data" refers to a device or method for playing the generated audio data so that the user can hear it.

[0487] A "means for receiving audio data and converting it into text data for analysis" is a device or method for receiving audio data, analyzing the data, and converting it into text data.

[0488] An "artificial intelligence-based means for generating an appropriate response" is a device or method that uses artificial intelligence techniques to generate an appropriate response based on analytical results and emotional data.

[0489] A "means for converting a generated response into voice data" is a device or method that converts an artificial intelligence-generated text response into voice data.

[0490] The "means for transmitting voice data" is a device or method for transmitting the generated voice data to the terminal.

[0491] "Communication means" refers to a device or method for transmitting and receiving voice data and emotion data between a server and a terminal.

[0492] A "means for generating a response based on analyzed text and emotional information" is a device or method for generating an appropriate response based on the analyzed text and emotional information.

[0493] The present invention provides a system for enabling people with hearing impairments to communicate smoothly in autonomous vehicles. This system includes speech recognition, emotion recognition, response generation, and speech synthesis, which are integrated and operate together.

[0494] System configuration

[0495] 1. Acquiring voice input

[0496] A microphone inside the vehicle captures the user's voice commands and is highly sensitive and has the ability to filter out interior noise.

[0497] 2. Voice Recognition

[0498] The captured voice data is sent from the vehicle's terminal to a server, which then converts the voice data into text using voice recognition software (e.g., Google Speech Recognition).

[0499] 3. Emotion recognition

[0500] The server uses the converted text data and the original voice data to input into an emotion recognition engine, which identifies the user's emotions from the tone of the voice and keywords used. This engine implements natural language processing and machine learning algorithms.

[0501] 4. Response Generation

[0502] The server generates an appropriate response based on the emotional information obtained by the emotion recognition engine. A generative AI model is used to generate the response. This AI model considers the emotional information and the content of the text data to propose the optimal response text.

[0503] 5. Speech Synthesis

[0504] The response text is converted to voice data on the server using Text-to-Speech (TTS) technology (e.g., pyttsx3), which mimics human voice tones and communicates to the user with natural pronunciation.

[0505] 6. Audio playback

[0506] The generated audio data is played back through the vehicle's speakers, allowing the user to hear the system's response.

[0507] Example of a system

[0508] For example, if a user in a self-driving vehicle says, "Turn left at the next intersection," the system will:

[0509] 1. Acquiring voice input

[0510] The user's instruction, "Turn left at the next intersection," is picked up by a microphone inside the car.

[0511] 2. Voice Recognition

[0512] The in-car device sends the voice data to the server, and Google Speech Recognition converts this data into text such as "Turn left at the next intersection."

[0513] 3. Emotion recognition

[0514] The server uses this text data and voice data to have an emotion recognition engine determine that the user is "confused."

[0515] 4. Response Generation

[0516] The server uses a generative AI model to generate a response based on emotion and text: "Okay, turn left at the next intersection."

[0517] 5. Speech Synthesis

[0518] The server uses text-to-speech technology to convert the generated response text into voice data.

[0519] 6. Audio playback

[0520] The response "Okay, turn left at the next intersection" is played over the car's speakers.

[0521] Prompt Sentence Examples

[0522] 001: "Turn left at the next intersection."

[0523] 002: System interprets the user's tone as confused and gives the response: "Okay. Turn left at the next intersection."

[0524] The system can accurately recognize a user's emotions and generate quick and appropriate responses based on those emotions, significantly improving communication for hearing-impaired users.

[0525] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0526] Step 1:

[0527] The user provides voice input, for example, "Turn left at the next intersection." The voice input is captured as digital voice data by a microphone inside the vehicle.

[0528] Step 2:

[0529] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol, and the server receives the data.

[0530] Step 3:

[0531] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech Recognition). For example, it generates text data such as "Turn left at the next intersection."

[0532] Step 4:

[0533] The server receives the text and voice data and inputs it into an emotion recognition engine, which uses natural language processing and machine learning algorithms to identify the user's emotion. For example, it outputs emotional information such as "confused."

[0534] Step 5:

[0535] The server uses a generative AI model based on emotion information and text data to generate an appropriate response. For example, in response to the emotion "confusion," it generates the response text "I understand. Turn left at the next intersection."

[0536] Step 6:

[0537] The server converts the generated response text into voice data using Text-to-Speech (TTS) technology (e.g., pyttsx3). For example, it generates voice data that says, "Okay, turn left at the next intersection."

[0538] Step 7:

[0539] The server sends the generated audio data to the device using an HTTP response, and the device receives the data.

[0540] Step 8:

[0541] The device then plays the received voice data through the car's speakers, allowing the user to hear the voice saying, "Okay, turn left at the next intersection."

[0542] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0543] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0544] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0545] [Second embodiment]

[0546] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0547] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0548] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0549] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0550] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0551] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0552] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0553] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0554] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0555] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0556] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0557] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0558] The present invention is a system for assisting people with hearing limitations in their daily communication by converting speech input into text data, generating appropriate responses, and converting them back into speech data for presentation to the user.

[0559] System configuration

[0560] 1. Acquiring voice input (device)

[0561] The device uses a microphone to receive user voice input, which is captured as digital audio data.

[0562] For example, a user might say, "Please give me a plastic bag at the convenience store."

[0563] 2. Sending voice data (terminal → server)

[0564] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[0565] 3. Speech recognition (server)

[0566] The server processes the received voice data and converts it into text data using OpenAI's Whisper API.

[0567] For example, speech data uttered "Please give me a plastic bag at the convenience store" is converted into text data stating "Please give me a plastic bag at the convenience store."

[0568] 4. Text analysis and response generation (server)

[0569] The server then uses artificial intelligence to analyze the converted text data, which allows it to understand the meaning of the text and generate an appropriate response.

[0570] For example, in response to the text data "Please give me a plastic bag at the convenience store," the AI ​​generates the response "Yes, would you like a plastic bag?"

[0571] 5. Speech synthesis (server)

[0572] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data.

[0573] The response text "Yes, would you like a plastic bag?" is converted into speech data.

[0574] 6. Sending audio data (server → device)

[0575] The server transmits the generated voice data to the terminal, which receives it and provides a response to the user.

[0576] 7. Audio playback (device)

[0577] The device then plays the received voice data through the speaker, and the user hears the voice saying, "Yes, would you like a plastic bag?"

[0578] Specific examples

[0579] Scenario: A user places an order at a cafe

[0580] 1. Acquiring voice input (device)

[0581] The user speaks to the terminal, "I'd like a small iced coffee, please."

[0582] The device picks up this audio and captures it as digital audio data.

[0583] 2. Sending voice data (terminal → server)

[0584] The terminal transmits the voice data to the server.

[0585] 3. Speech recognition (server)

[0586] The server converts the speech data into text "I'd like a small iced coffee please."

[0587] 4. Text analysis and response generation (server)

[0588] The server uses artificial intelligence to analyze the text data and generate an appropriate response: "Thank you. Is small size okay?"

[0589] 5. Speech synthesis (server)

[0590] The server converts the response text into audio data.

[0591] 6. Sending audio data (server → device)

[0592] The server transmits the audio data to the terminal.

[0593] 7. Audio playback (device)

[0594] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[0595] In this way, the present invention supports everyday communication for people with hearing limitations through a series of processes that convert voice data into text data, generate an appropriate response, and provide it to the user again as voice data.

[0596] The processing flow will be explained below.

[0597] Step 1:

[0598] The device activates the microphone and receives voice input, for example, when the user says, "I'd like a coffee, please."

[0599] Step 2:

[0600] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[0601] Step 3:

[0602] The terminal uses an HTTP request to send the audio data to the server.

[0603] Step 4:

[0604] The server analyzes the received voice data and converts it into text data by calling a voice recognition API (e.g. Whisper API). For example, the voice saying "One coffee please" is converted into the text "One coffee please."

[0605] Step 5:

[0606] The server sends the text data to an artificial intelligence system that analyzes it to generate an appropriate response, such as "Hot or Ice?"

[0607] Step 6:

[0608] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into speech data. For example, the text "Is it hot or iced?" is converted into speech data.

[0609] Step 7:

[0610] The server packages the generated audio data and prepares an HTTP response to send to the device.

[0611] Step 8:

[0612] The server uses an HTTP response to send the audio data to the terminal.

[0613] Step 9:

[0614] The device decodes the received audio data and plays it through the speaker.

[0615] Step 10:

[0616] The user hears a voice response saying, "Hot or iced?"

[0617] The above is the specific program processing flow of the system, which allows users with hearing limitations to communicate smoothly.

[0618] Example 1

[0619] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] It is difficult for people with hearing limitations to communicate smoothly in their daily lives. Current communication support systems convert voice input into text, generate an appropriate response, and then convert it back into voice to provide to the user. However, these systems often lack the quality, immediacy, and appropriate response generation of the voice data. Processing delays and unstable communication are also problems. A system that can solve these issues and provide fast, high-quality communication support is needed.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0622] In this invention, the server includes a preprocessing means for converting a voice input into digital voice data and generating high-quality voice data, a communication means for transmitting the high-quality voice data to an analysis server, a means for converting the high-quality voice data into text data, a means for analyzing the text data and generating an appropriate response text, a means for converting the response text into voice data, and a means for receiving and playing the voice data, thereby enabling the generation of high-quality voice data and instantaneous response generation.

[0623] "Audio input" refers to words or sounds made by a user through a microphone or other sound capturing device.

[0624] "Digital audio data" means data that has been converted from analog audio into a digital signal for further processing or communication.

[0625] The "pre-processing means" refers to a means for performing processes such as noise removal and echo cancellation when converting captured audio into high-quality digital audio data.

[0626] "Communication means" refers to the means for sending and receiving data between the terminal and the server. Typically, the HTTP protocol or a dedicated protocol is used via the Internet.

[0627] "Text data" refers to data in the form of a string of characters obtained by analyzing voice data. This data is in a format that is easy for computers to process.

[0628] "Analysis means" refers to artificial intelligence or algorithms that understand the content of text data and generate appropriate responses.

[0629] "Appropriate response text" refers to text data in response to text data from a user that is generated by an AI model, etc.

[0630] "Means for converting into voice data" refers to Text-to-Speech (TTS) technology or APIs for converting text data back into voice data.

[0631] The "means for playing" refers to a means for allowing the user to listen to the generated audio data through a speaker, earphones, or the like.

[0632] The present invention provides a system for supporting communication for users with hearing limitations. This system converts voice input into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. The following describes an embodiment of the present invention based on specific processing and examples.

[0633] Hardware and software used

[0634] 1. Terminal

[0635] Hardware: Microphone, speaker, communication module (Wi-Fi, 4G / 5G module, etc.)

[0636] Software: Audio capture app, digital signal processing software, communication protocol stack

[0637] 2. Server

[0638] Hardware: High-performance processor, memory, and large storage capacity

[0639] Software: Speech recognition APIs (e.g., Whisper API), text analysis engines, generative AI models (e.g., GPT-3), Text-to-Speech (TTS) APIs

[0640] Processing Description

[0641] 1. Acquiring voice input

[0642] The device uses a microphone to capture the user's voice input as digital audio data, which is then pre-processed with noise reduction and echo cancellation before being stored in a high-quality format and sent directly to the server.

[0643] 2. Sending audio data

[0644] The device then sends the captured high-quality audio data to the server, usually via HTTP requests, but sometimes using proprietary communication protocols.

[0645] 3. Voice Recognition

[0646] The server converts the received voice data into text data using OpenAI's Whisper API, which analyzes the voice data and converts it into a string format, allowing it to accurately transcribe what the user is saying.

[0647] 4. Text Analysis and Response Generation

[0648] The server analyzes the converted text data and generates an appropriate response using a generative AI model (e.g., GPT-3), which understands the user's intent and creates a response that is context-sensitive.

[0649] 5. Speech Synthesis

[0650] The server then sends the generated response text to a Text-to-Speech (TTS) API, which converts the response into audio data, possibly including selecting a voice narrator and setting emotional expressions.

[0651] 6. Sending audio data

[0652] The server then sends the generated audio data back to the device, usually via an HTTP request.

[0653] 7. Audio playback

[0654] The terminal plays the received voice data through the speaker, allowing the user to hear the response from the system.

[0655] Specific examples

[0656] Scenario: A user places an order at a cafe

[0657] Acquiring voice input

[0658] The user speaks to the device, saying, "I'd like a small iced coffee, please." The device captures this speech as high-quality digital audio data.

[0659] Sending audio data

[0660] The device transmits the captured audio data to the server.

[0661] Voice Recognition

[0662] The server uses the Whisper API to convert the received voice data into text, "I'd like a small iced coffee, please."

[0663] Text analysis and response generation

[0664] The server sends a prompt to the generative AI model to generate the appropriate response: "Thank you. Would you like the small size?"

[0665] Speech synthesis

[0666] The server converts this response text into voice data using a TTS API.

[0667] Sending audio data

[0668] The server transmits the generated voice data to the terminal.

[0669] Playing audio

[0670] The terminal plays the received audio data, and the user can hear, "Thank you. Is small size OK?"

[0671] Prompt Sentence Examples

[0672] prompt:

[0673] "If the user says, 'I'd like a small iced coffee, please,' generate an appropriate response."

[0674] As described above, the present invention makes it possible to maintain the quality of digital voice data and generate instantaneous responses, thereby smoothly supporting everyday communication for users with hearing limitations.

[0675] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0676] Step 1:

[0677] The device captures the user's voice input as digital voice data. For example, when the user says, "I'd like a small iced coffee, please," the microphone picks up this voice and converts the analog signal into a digital signal. The input is the user's voice, and the output is digital voice data.

[0678] Step 2:

[0679] The device preprocesses the captured digital audio data. This preprocessing includes noise reduction, echo cancellation, and audio clipping. This results in high-quality audio data. The input is digital audio data, and the output is preprocessed, high-quality digital audio data.

[0680] Step 3:

[0681] The terminal sends the preprocessed audio data to the server. The communication protocol is typically an HTTP POST request. The input is high-quality digital audio data, and the output is a response to the server confirming successful data transmission.

[0682] Step 4:

[0683] The server converts the received voice data into text data using OpenAI's Whisper API. This API analyzes the voice data and converts the user's speech into a string format. The input is high-quality digital voice data, and the output is text data. For example, "I'd like a small iced coffee, please." is converted into text.

[0684] Step 5:

[0685] The server analyzes the text data and generates an appropriate response. A generative AI model (e.g., GPT-3) is used for this analysis and response generation. The input is text data, and the output is an appropriate response text. For example, in response to "I'd like a small iced coffee, please," the server generates the response "Thank you. Is the small size okay?"

[0686] Step 6:

[0687] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts it into voice data. This API converts text data into a voice signal. The input is the response text, and the output is voice data. For example, "Thank you. Is small size okay?" is converted into voice data.

[0688] Step 7:

[0689] The server sends the generated audio data to the device. Again, an HTTP POST request is used. The input is the audio data, and the output is a response indicating successful data transmission to the device.

[0690] Step 8:

[0691] The device then plays the received audio data to the user. At this time, the audio is output using a speaker or earphones. The input is the audio data, and the output is the audio heard by the user. Specifically, the device plays back "Thank you. Is small size OK?"

[0692] (Application example 1)

[0693] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0694] People with hearing limitations often face significant limitations in their daily lives, especially when communicating in brick-and-mortar stores. For example, when communicating with store clerks about orders, questions, or confirmations, they may find it difficult to hear what the other person is saying, hindering smooth communication. There is a need for technology that can solve these problems and enable people with hearing limitations to communicate smoothly in brick-and-mortar stores.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0696] In this invention, the server includes means for acquiring a voice input, means for converting the voice input into text data, means for analyzing the text data and generating an appropriate response text, means for converting the response text into voice data, means for playing the voice data, means including an artificial intelligence model for processing the text data and the voice data, means for the system to be executed on a smart device (smartphone, smart glasses, etc.) installed in a physical store, and means for the voice data to facilitate communication in the physical store, thereby enabling people with hearing limitations to smoothly communicate in the physical store, such as when placing an order or asking a question.

[0697] "Means for acquiring voice input" refers to a device or function for capturing voice uttered by a user and inputting it into the system.

[0698] The "means for converting into text data" refers to a device or function for converting acquired voice data into text data.

[0699] The "means for generating an appropriate response text" refers to a device or function for analyzing the converted text data and generating an appropriate response based on the analyzed text data.

[0700] The "means for converting into voice data" refers to a device or function for converting the generated response text into voice data.

[0701] "Means for playing audio data" refers to the function of using a speaker or other playback device to provide the converted audio data to the user.

[0702] "Means including an artificial intelligence model" refers to a device or system function that incorporates artificial intelligence used to analyze text data and generate responses.

[0703] A "smart device" is a portable device, such as a smartphone or smart glasses, that has Internet connectivity and application execution capabilities and is used as a user interface.

[0704] "Means to be installed in a physical store" refers to the means by which a system with functions such as language and voice processing, data transmission and reception, etc. can be operated within a physical store.

[0705] "Means aimed at facilitating communication in physical stores" refers to the functions of devices and systems that improve and support everyday communication such as customer service and transactions in physical stores.

[0706] The present invention provides a system that supports people with hearing impairments to smoothly communicate in physical stores. Specific embodiments are described below in detail.

[0707] System program generation

[0708] This system supports communication through a series of processes: it receives voice input, converts it into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. Specifically, there are three main entities involved in this system: the server, the terminal, and the user.

[0709] Hardware and Software Configuration

[0710] Hardware:

[0711] Portable devices such as smartphones and smart glasses

[0712] Server (cloud-based, e.g. AWS, Google Cloud)

[0713] software:

[0714] Speech Recognition API (OpenAI's Whisper)

[0715] Text analysis and response generation model (OpenAI's GPT-4)

[0716] Speech synthesis API (Google Text-to-Speech)

[0717] Processing flow and data processing

[0718] Getting user voice input

[0719] The device captures the user's voice through the microphone. For example, the user says, "One espresso, please."

[0720] Sending audio data

[0721] The device sends the captured audio data to the server using the HTTP protocol.

[0722] Voice Recognition

[0723] The server uses OpenAI's Whisper API to convert voice data into text. For example, "One espresso please" is converted into text "One espresso please."

[0724] Text analysis and response generation

[0725] The server uses a GPT-4 model to analyze the text data and generate an appropriate response. For example, in response to "One espresso please," the server generates the response "Okay, one espresso please."

[0726] Speech synthesis

[0727] The generated response text is converted into audio data using the Google Text-to-Speech API.

[0728] Sending and playing audio data

[0729] The server sends the generated voice data to the terminal, which plays it back. The user hears the voice saying, "Okay, one espresso, please."

[0730] Specific examples

[0731] This system, which uses a combination of a server and a terminal, is applied as follows:

[0732] Case 1: Ordering at a cafe

[0733] When a user places an order at a cafe, they speak into their smartphone, saying, "One espresso, please." The device recognizes the speech and sends it to the server. The server converts it to text, generates an appropriate response, converts it to audio data, and sends it back to the device. The device then plays back the response, allowing the user to hear it and confirm that their order was received correctly.

[0734] Prompt Sentence Examples

[0735] User: One espresso please.

[0736] AI Response: Ok, one espresso please.

[0737] In this way, the system of the present invention provides a specific embodiment that enables people with hearing limitations to smoothly communicate in physical stores.

[0738] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0739] Step 1:

[0740] Getting user voice input

[0741] Subject: Terminal

[0742] Specific behavior:

[0743] The device captures the user's voice through the built-in microphone.

[0744] Input: User speech (e.g., user says "One espresso please").

[0745] Data processing: The microphone converts analog audio into digital audio data.

[0746] Output: Digital audio data.

[0747] Step 2:

[0748] Sending audio data

[0749] Subject: Terminal

[0750] Specific behavior:

[0751] The device sends the captured audio data to the server.

[0752] Input: Digital audio data.

[0753] Data Calculation: Send data to the server using an HTTP request.

[0754] Output: The audio data sent to the server.

[0755] Step 3:

[0756] Voice Recognition

[0757] Subject: Server

[0758] Specific behavior:

[0759] The server converts the received voice data into text using OpenAI's Whisper API.

[0760] Input: Received audio data.

[0761] Data processing: Voice data is analyzed using the Whisper API and converted into text data.

[0762] Output: Text data (e.g., "One espresso, please").

[0763] Step 4:

[0764] Text analysis and response generation

[0765] Subject: Server

[0766] Specific behavior:

[0767] The server uses the GPT-4 model to analyze the text data and generate an appropriate response.

[0768] Input: Text data ("One espresso please").

[0769] Data computation: Semantic interpretation and response generation using the GPT-4 model.

[0770] Output: The response text data (e.g., "Okay, one espresso, please.").

[0771] Step 5:

[0772] Speech synthesis

[0773] Subject: Server

[0774] Specific behavior:

[0775] The server converts the generated response text into audio data using the Google Text-to-Speech API.

[0776] Input: Response text data.

[0777] Data processing: Convert text data into audio data using the Text-to-Speech API.

[0778] Output: Response voice data (e.g., "Okay, one espresso, please").

[0779] Step 6:

[0780] Sending audio data

[0781] Subject: Server

[0782] Specific behavior:

[0783] The server transmits the generated voice data to the terminal.

[0784] Input: Response audio data.

[0785] Data calculation: The audio data is returned to the terminal as an HTTP response.

[0786] Output: The audio data sent to the device.

[0787] Step 7:

[0788] Playing audio

[0789] Subject: Terminal

[0790] Specific behavior:

[0791] The audio data received by the terminal is played back through the speaker.

[0792] Input: Received audio data.

[0793] Data processing: The speaker converts digital audio data into an analog signal.

[0794] Output: Analogue sound signal (e.g. user hears the voice "Okay, one espresso please").

[0795] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0796] This invention is a system for enabling people with hearing limitations to communicate effectively in everyday life, and is configured with an emotion engine. This system converts voice input into text data, generates an appropriate response, and converts it back into voice data to provide to the user.

[0797] System configuration

[0798] 1. Acquiring voice input (device)

[0799] The device activates the microphone and receives the user's voice input. For example, the user might say, "Please give me a plastic bag at the convenience store." This voice input is captured as digital audio data.

[0800] 2. Sending voice data (terminal → server)

[0801] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[0802] 3. Speech recognition (server)

[0803] The server receives the voice data and converts it into text data using a speech recognition API. For example, the voice saying "Please give me a plastic bag at the convenience store" is converted into text data "Please give me a plastic bag at the convenience store."

[0804] 4. Emotion Recognition (Server)

[0805] The server inputs the text data and voice data into an emotion engine to recognize the user's emotions. For example, it recognizes the user's emotions such as "I'm in a hurry" or "I'm in a hurry" from the tone of the voice and the content of the text.

[0806] 5. Text Analysis and Response Generation (Server)

[0807] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[0808] 6. Speech synthesis (server)

[0809] The server sends the generated response text to a Text-to-Speech (TTS) API and converts it into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[0810] 7. Sending audio data (server → device)

[0811] The server packages the generated audio data and prepares an HTTP response to send to the device.

[0812] 8. Audio playback (device)

[0813] The device decodes the received voice data and plays it through the speaker, and the user can hear the voice saying "It will be ready soon."

[0814] Specific examples

[0815] Scenario: A user places an order at a cafe

[0816] 1. Acquiring voice input (device)

[0817] The user speaks to the terminal, "I'd like a small iced coffee, please."

[0818] The device picks up this audio and captures it as digital audio data.

[0819] 2. Sending voice data (terminal → server)

[0820] The terminal transmits the voice data to the server.

[0821] 3. Speech recognition (server)

[0822] The server converts the speech data into text "I'd like a small iced coffee please."

[0823] 4. Emotion Recognition (Server)

[0824] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[0825] 5. Text Analysis and Response Generation (Server)

[0826] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[0827] 6. Speech synthesis (server)

[0828] The server converts the response text into audio data.

[0829] 7. Sending audio data (server → device)

[0830] The server transmits the audio data to the terminal.

[0831] 8. Audio playback (device)

[0832] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[0833] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate corresponding appropriate responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[0834] The processing flow will be explained below.

[0835] Step 1:

[0836] The device activates the microphone and receives the user's voice input, for example, the user says, "I'd like a coffee, please."

[0837] Step 2:

[0838] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[0839] Step 3:

[0840] The terminal uses an HTTP request to send the audio data to the server.

[0841] Step 4:

[0842] The server analyzes the received voice data and converts it into text data by calling a speech recognition API (e.g., Whisper API). For example, the speech "One coffee please" is converted into the text "One coffee please."

[0843] Step 5:

[0844] The server inputs the text and voice data into an emotion engine to recognize the user's emotions. From the tone of the voice and the content of the text, the server recognizes emotions such as "I'm in a hurry" or "I'm in a hurry."

[0845] Step 6:

[0846] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[0847] Step 7:

[0848] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[0849] Step 8:

[0850] The server packages the generated audio data and prepares an HTTP response to send to the device.

[0851] Step 9:

[0852] The server uses an HTTP response to send the audio data to the terminal.

[0853] Step 10:

[0854] The device decodes the received audio data and plays it through the speaker.

[0855] Step 11:

[0856] The user hears the response, "It will be ready shortly."

[0857] Specific examples

[0858] Scenario: A user places an order at a cafe

[0859] Step 1:

[0860] The user speaks to the terminal, "I'd like a small iced coffee, please."

[0861] Step 2:

[0862] The device picks up this audio and captures it as digital audio data.

[0863] Step 3:

[0864] The terminal transmits the voice data to the server.

[0865] Step 4:

[0866] The server converts the speech data into text "I'd like a small iced coffee please."

[0867] Step 5:

[0868] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[0869] Step 6:

[0870] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[0871] Step 7:

[0872] The server converts the response text into audio data.

[0873] Step 8:

[0874] The server transmits the audio data to the terminal.

[0875] Step 9:

[0876] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[0877] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate appropriate corresponding responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[0878] Example 2

[0879] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0880] Conventional speech recognition systems can only convert speech data into text and generate responses, making it difficult to accurately grasp a user's emotions and intentions. This results in a poor user experience, particularly for people with hearing limitations. Furthermore, because they do not recognize emotions, it is difficult to provide appropriate responses based on the user's situation and feelings. As a result, communication is ineffective, often causing difficulties in daily life.

[0881] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice input into text data, a means for analyzing the text data and voice data and recognizing the user's emotion, and a means for generating an appropriate response text based on the emotion information. This makes it possible to accurately recognize the user's emotion and provide an appropriate response corresponding to that emotion.

[0882] "Audio input" refers to a voice signal emitted by a user into a device, which is captured as digital data.

[0883] "Text data" refers to text information converted from voice input using voice recognition.

[0884] "Audio data" refers to data in which an audio signal is expressed in digital form.

[0885] "Emotion recognition" refers to the process of analyzing speech and text data to identify a user's emotional state.

[0886] "Response text" refers to a textual representation of an appropriate response generated based on the user's utterance and their sentiment.

[0887] "Speech synthesis" refers to the technical process of converting text data into speech data.

[0888] "Playback" refers to audio data being provided to a user in an audible form through an output device such as a speaker.

[0889] "Transmitting and receiving" refers to the process of transferring data from one device to another.

[0890] This invention is a system that enables people with hearing limitations to communicate effectively in their daily lives, and performs a series of processes including voice data acquisition, analysis, emotion recognition, response generation, and voice playback.

[0891] Hardware and software used

[0892] Hardware:

[0893] Device: A device with a microphone and speaker, such as a smartphone or voice assistant device.

[0894] Server: Cloud environment (e.g. AWS, Google Cloud, Azure)

[0895] software:

[0896] Speech Recognition API: Google Cloud Speech-to-Text

[0897] Emotion Recognition Engine: Software that implements a specific emotion recognition algorithm (e.g., a proprietary machine learning model).

[0898] Text-to-Speech (TTS) API: Amazon Polly, etc.

[0899] System Operation Overview

[0900] 1. Acquiring voice input:

[0901] The device activates a microphone and digitally captures what the user says.

[0902] 2. Sending audio data:

[0903] The device sends the captured audio data to the server via an HTTP request.

[0904] 3. Speech Recognition:

[0905] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data.

[0906] 4. Emotion recognition:

[0907] The server inputs the text data obtained by speech recognition and the original speech data into an emotion recognition engine to recognize the user's emotions, such as "tired" or "in a hurry."

[0908] 5. Text analysis and response generation:

[0909] Based on the emotion information, the server generates an appropriate response text.

[0910] 6. Speech synthesis:

[0911] The server sends the generated response text to a Text-to-Speech API such as Amazon Polly, which converts the response into audio data.

[0912] 7. Sending audio data:

[0913] The server sends the generated audio data to the terminal as an HTTP response.

[0914] 8. Audio playback:

[0915] The terminal decodes the received audio data and plays it back to the user through the speaker.

[0916] Specific examples

[0917] Scenario: A user places an order at a cafe

[0918] For example, if a user says to their smartphone, "Please give me a small iced coffee," the system will operate as follows:

[0919] 1. Acquiring voice input:

[0920] The device captures the audio and stores it as digital audio data.

[0921] 2. Sending audio data:

[0922] The terminal transmits the voice data to the server.

[0923] 3. Speech Recognition:

[0924] The server sends the voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "I'd like a small iced coffee, please."

[0925] 4. Emotion recognition:

[0926] The server analyzes the text data and voice data and recognizes the user's emotion as "calm."

[0927] 5. Text analysis and response generation:

[0928] Based on the emotion information, the server generates a response text saying "Thank you. Is small size OK?"

[0929] 6. Speech synthesis:

[0930] The server sends this response text to Amazon Polly, which converts it into voice data.

[0931] 7. Sending audio data:

[0932] The server transmits the generated voice data to the terminal.

[0933] 8. Audio playback:

[0934] The terminal plays the audio data, and the user hears "Thank you. Is small size OK?"

[0935] Prompt Sentence Examples

[0936] Please explain a specific scenario in which a user orders a small iced coffee at a cafe, and provide a detailed explanation of each process: speech recognition, emotion recognition, response generation, speech synthesis, and speech playback.

[0937] The system enables people with hearing limitations to communicate more appropriately and effectively through emotion recognition technology.

[0938] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0939] Step 1: Getting voice input (device)

[0940] Input: User's voice

[0941] Processing: The device activates the microphone and captures the user's voice input as digital audio data.

[0942] Output: Digital audio data

[0943] Specific behavior:

[0944] The device activates the microphone and waits for the user's voice input.

[0945] A user says, "I'd like a small iced coffee at the cafe."

[0946] The device captures this audio digitally and stores it temporarily in memory.

[0947] Step 2: Sending voice data (device → server)

[0948] Input: Digital audio data

[0949] Processing: The device sends the captured audio data to the server via an HTTP POST request, using HTTPS for security reasons.

[0950] Output: Audio data sent to the server

[0951] Specific behavior:

[0952] The audio data is included in the body of the HTTPS request.

[0953] Sends a request to a specific endpoint on the server.

[0954] Step 3: Speech Recognition (Server)

[0955] Input: Audio data sent to the server

[0956] Processing: The server inputs the received audio data into the Google Cloud Speech-to-Text API and converts the audio into text data.

[0957] Output: Text data

[0958] Specific behavior:

[0959] Send the audio data to the Google Cloud Speech-to-Text API.

[0960] Receives the converted text data and passes it to the next process.

[0961] For example, the speech "I'd like a small iced coffee at the cafe, please" is converted into the text "I'd like a small iced coffee at the cafe, please."

[0962] Step 4: Emotion Recognition (Server)

[0963] Input: Text data and audio data

[0964] Processing: The server inputs the text and voice data obtained through speech recognition into an emotion recognition engine to recognize the user's emotions. For example, it may determine that the user is "calm" based on the tone of the voice and the way the words are spoken.

[0965] Output: User's emotional state

[0966] Specific behavior:

[0967] Text data and voice data are input into the emotion recognition engine.

[0968] The emotion information obtained from the emotion recognition engine is passed to the next process.

[0969] For example, the emotion "calm" is identified.

[0970] Step 5: Text analysis and response generation (server)

[0971] Input: Emotion information and text data

[0972] Processing: Based on the emotion information, the server generates an appropriate response text. For example, if the user is recognized as "calm", the response will be "Thank you. Is small size OK?"

[0973] Output: Response text

[0974] Specific behavior:

[0975] The text data and sentiment information are analyzed to determine an appropriate response text.

[0976] Generates response text and passes it to the next process.

[0977] Step 6: Speech synthesis (server)

[0978] Input: Response text

[0979] Processing: The server sends the response text to a Text-to-Speech (TTS) API (e.g., Amazon Polly) and converts it into audio data.

[0980] Output: Audio data

[0981] Specific behavior:

[0982] Send the response text to the TTS API.

[0983] The received audio data is passed to the next process.

[0984] Step 7: Sending audio data (server → device)

[0985] Input: Generated audio data

[0986] Processing: The server sends the generated audio data to the device in an HTTP response. HTTPS is used to ensure security.

[0987] Output: Audio data sent to the device

[0988] Specific behavior:

[0989] The audio data is included in the body of the HTTPS response.

[0990] Sends a response to a specific endpoint on the device.

[0991] Step 8: Playing Audio (Device)

[0992] Input: Audio data sent to the device

[0993] Processing: The device decodes the received audio data and plays it through the speaker. The user can hear the audio "Thank you. Is small size OK?"

[0994] Output: The audio the user can hear

[0995] Specific behavior:

[0996] Decodes audio data and converts it into a playable format.

[0997] Play audio data through a speaker.

[0998] The above is the specific processing flow of this system.

[0999] (Application example 2)

[1000] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1001] It is difficult for people with hearing limitations to communicate effectively in everyday life. In particular, when interacting with complex systems such as self-driving vehicles, it is necessary to generate appropriate responses that take into account the user's emotions. However, current technology has difficulty not only accepting voice input but also accurately recognizing the user's emotions and generating responses based on those emotions, resulting in a decline in the quality of communication.

[1002] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion from voice data and text data, means for generating an appropriate response text taking the emotion into consideration, and means for converting the generated response text into voice data. This makes it possible to recognize the user's emotion and generate an appropriate and prompt response based on the emotion.

[1003] A "means for obtaining voice input" is a device or method for obtaining voice from a user as digital voice data.

[1004] A "means for converting speech input into text data" is a device or method for analyzing acquired speech input and converting it into corresponding text data.

[1005] The "means for recognizing emotions" is a device or method for identifying a user's emotions from text data and voice data.

[1006] The "means for analyzing text data and generating an appropriate response text" is a device or method for analyzing text data in consideration of emotional information and generating an appropriate response.

[1007] The "means for converting response text into voice data" is a device or method for converting the generated response text into voice data.

[1008] The "means for playing audio data" refers to a device or method for playing the generated audio data so that the user can hear it.

[1009] A "means for receiving audio data and converting it into text data for analysis" is a device or method for receiving audio data, analyzing the data, and converting it into text data.

[1010] An "artificial intelligence-based means for generating an appropriate response" is a device or method that uses artificial intelligence techniques to generate an appropriate response based on analytical results and emotional data.

[1011] A "means for converting a generated response into voice data" is a device or method that converts an artificial intelligence-generated text response into voice data.

[1012] The "means for transmitting voice data" is a device or method for transmitting the generated voice data to the terminal.

[1013] "Communication means" refers to a device or method for transmitting and receiving voice data and emotion data between a server and a terminal.

[1014] A "means for generating a response based on analyzed text and emotional information" is a device or method for generating an appropriate response based on the analyzed text and emotional information.

[1015] The present invention provides a system for enabling people with hearing impairments to communicate smoothly in autonomous vehicles. This system includes speech recognition, emotion recognition, response generation, and speech synthesis, which are integrated and operate together.

[1016] System configuration

[1017] 1. Acquiring voice input

[1018] A microphone inside the vehicle captures the user's voice commands and is highly sensitive and has the ability to filter out interior noise.

[1019] 2. Voice Recognition

[1020] The captured voice data is sent from the vehicle's terminal to a server, which then converts the voice data into text using voice recognition software (e.g., Google Speech Recognition).

[1021] 3. Emotion recognition

[1022] The server uses the converted text data and the original voice data to input into an emotion recognition engine, which identifies the user's emotions from the tone of the voice and keywords used. This engine implements natural language processing and machine learning algorithms.

[1023] 4. Response Generation

[1024] The server generates an appropriate response based on the emotional information obtained by the emotion recognition engine. A generative AI model is used to generate the response. This AI model considers the emotional information and the content of the text data to propose the optimal response text.

[1025] 5. Speech Synthesis

[1026] The response text is converted to voice data on the server using Text-to-Speech (TTS) technology (e.g., pyttsx3), which mimics human voice tones and communicates to the user with natural pronunciation.

[1027] 6. Audio playback

[1028] The generated audio data is played back through the vehicle's speakers, allowing the user to hear the system's response.

[1029] Example of a system

[1030] For example, if a user in a self-driving vehicle says, "Turn left at the next intersection," the system will:

[1031] 1. Acquiring voice input

[1032] The user's instruction, "Turn left at the next intersection," is picked up by a microphone inside the car.

[1033] 2. Voice Recognition

[1034] The in-car device sends the voice data to the server, and Google Speech Recognition converts this data into text such as "Turn left at the next intersection."

[1035] 3. Emotion recognition

[1036] The server uses this text data and voice data to have an emotion recognition engine determine that the user is "confused."

[1037] 4. Response Generation

[1038] The server uses a generative AI model to generate a response based on emotion and text: "Okay, turn left at the next intersection."

[1039] 5. Speech Synthesis

[1040] The server uses text-to-speech technology to convert the generated response text into voice data.

[1041] 6. Audio playback

[1042] The response "Okay, turn left at the next intersection" is played over the car's speakers.

[1043] Prompt Sentence Examples

[1044] 001: "Turn left at the next intersection."

[1045] 002: System interprets the user's tone as confused and gives the response: "Okay. Turn left at the next intersection."

[1046] The system can accurately recognize a user's emotions and generate quick and appropriate responses based on those emotions, significantly improving communication for hearing-impaired users.

[1047] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1048] Step 1:

[1049] The user provides voice input, for example, "Turn left at the next intersection." The voice input is captured as digital voice data by a microphone inside the vehicle.

[1050] Step 2:

[1051] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol, and the server receives the data.

[1052] Step 3:

[1053] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech Recognition). For example, it generates text data such as "Turn left at the next intersection."

[1054] Step 4:

[1055] The server receives the text and voice data and inputs it into an emotion recognition engine, which uses natural language processing and machine learning algorithms to identify the user's emotion. For example, it outputs emotional information such as "confused."

[1056] Step 5:

[1057] The server uses a generative AI model based on emotion information and text data to generate an appropriate response. For example, in response to the emotion "confusion," it generates the response text "I understand. Turn left at the next intersection."

[1058] Step 6:

[1059] The server converts the generated response text into voice data using Text-to-Speech (TTS) technology (e.g., pyttsx3). For example, it generates voice data that says, "Okay, turn left at the next intersection."

[1060] Step 7:

[1061] The server sends the generated audio data to the device using an HTTP response, and the device receives the data.

[1062] Step 8:

[1063] The device then plays the received voice data through the car's speakers, allowing the user to hear the voice saying, "Okay, turn left at the next intersection."

[1064] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1065] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1066] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1067] [Third embodiment]

[1068] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1069] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1070] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1071] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1072] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1073] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1074] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1075] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1076] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1077] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1078] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1079] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1080] The present invention is a system for assisting people with hearing limitations in their daily communication by converting speech input into text data, generating appropriate responses, and converting them back into speech data for presentation to the user.

[1081] System configuration

[1082] 1. Acquiring voice input (device)

[1083] The device uses a microphone to receive user voice input, which is captured as digital audio data.

[1084] For example, a user might say, "Please give me a plastic bag at the convenience store."

[1085] 2. Sending voice data (terminal → server)

[1086] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[1087] 3. Speech recognition (server)

[1088] The server processes the received voice data and converts it into text data using OpenAI's Whisper API.

[1089] For example, speech data uttered "Please give me a plastic bag at the convenience store" is converted into text data stating "Please give me a plastic bag at the convenience store."

[1090] 4. Text analysis and response generation (server)

[1091] The server then uses artificial intelligence to analyze the converted text data, which allows it to understand the meaning of the text and generate an appropriate response.

[1092] For example, in response to the text data "Please give me a plastic bag at the convenience store," the AI ​​generates the response "Yes, would you like a plastic bag?"

[1093] 5. Speech synthesis (server)

[1094] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data.

[1095] The response text "Yes, would you like a plastic bag?" is converted into speech data.

[1096] 6. Sending audio data (server → device)

[1097] The server transmits the generated voice data to the terminal, which receives it and provides a response to the user.

[1098] 7. Audio playback (device)

[1099] The device then plays the received voice data through the speaker, and the user hears the voice saying, "Yes, would you like a plastic bag?"

[1100] Specific examples

[1101] Scenario: A user places an order at a cafe

[1102] 1. Acquiring voice input (device)

[1103] The user speaks to the terminal, "I'd like a small iced coffee, please."

[1104] The device picks up this audio and captures it as digital audio data.

[1105] 2. Sending voice data (terminal → server)

[1106] The terminal transmits the voice data to the server.

[1107] 3. Speech recognition (server)

[1108] The server converts the speech data into text "I'd like a small iced coffee please."

[1109] 4. Text analysis and response generation (server)

[1110] The server uses artificial intelligence to analyze the text data and generate an appropriate response: "Thank you. Is small size okay?"

[1111] 5. Speech synthesis (server)

[1112] The server converts the response text into audio data.

[1113] 6. Sending audio data (server → device)

[1114] The server transmits the audio data to the terminal.

[1115] 7. Audio playback (device)

[1116] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[1117] In this way, the present invention supports everyday communication for people with hearing limitations through a series of processes that convert voice data into text data, generate an appropriate response, and provide it to the user again as voice data.

[1118] The processing flow will be explained below.

[1119] Step 1:

[1120] The device activates the microphone and receives voice input, for example, when the user says, "I'd like a coffee, please."

[1121] Step 2:

[1122] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[1123] Step 3:

[1124] The terminal uses an HTTP request to send the audio data to the server.

[1125] Step 4:

[1126] The server analyzes the received voice data and converts it into text data by calling a voice recognition API (e.g. Whisper API). For example, the voice saying "One coffee please" is converted into the text "One coffee please."

[1127] Step 5:

[1128] The server sends the text data to an artificial intelligence system that analyzes it to generate an appropriate response, such as "Hot or Ice?"

[1129] Step 6:

[1130] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into speech data. For example, the text "Is it hot or iced?" is converted into speech data.

[1131] Step 7:

[1132] The server packages the generated audio data and prepares an HTTP response to send to the device.

[1133] Step 8:

[1134] The server uses an HTTP response to send the audio data to the terminal.

[1135] Step 9:

[1136] The device decodes the received audio data and plays it through the speaker.

[1137] Step 10:

[1138] The user hears a voice response saying, "Hot or iced?"

[1139] The above is the specific program processing flow of the system, which allows users with hearing limitations to communicate smoothly.

[1140] Example 1

[1141] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1142] It is difficult for people with hearing limitations to communicate smoothly in their daily lives. Current communication support systems convert voice input into text, generate an appropriate response, and then convert it back into voice to provide to the user. However, these systems often lack the quality, immediacy, and appropriate response generation of the voice data. Processing delays and unstable communication are also problems. A system that can solve these issues and provide fast, high-quality communication support is needed.

[1143] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1144] In this invention, the server includes a preprocessing means for converting a voice input into digital voice data and generating high-quality voice data, a communication means for transmitting the high-quality voice data to an analysis server, a means for converting the high-quality voice data into text data, a means for analyzing the text data and generating an appropriate response text, a means for converting the response text into voice data, and a means for receiving and playing the voice data, thereby enabling the generation of high-quality voice data and instantaneous response generation.

[1145] "Audio input" refers to words or sounds made by a user through a microphone or other sound capturing device.

[1146] "Digital audio data" means data that has been converted from analog audio into a digital signal for further processing or communication.

[1147] The "pre-processing means" refers to a means for performing processes such as noise removal and echo cancellation when converting captured audio into high-quality digital audio data.

[1148] "Communication means" refers to the means for sending and receiving data between the terminal and the server. Typically, the HTTP protocol or a dedicated protocol is used via the Internet.

[1149] "Text data" refers to data in the form of a string of characters obtained by analyzing voice data. This data is in a format that is easy for computers to process.

[1150] "Analysis means" refers to artificial intelligence or algorithms that understand the content of text data and generate appropriate responses.

[1151] "Appropriate response text" refers to text data in response to text data from a user that is generated by an AI model, etc.

[1152] "Means for converting into voice data" refers to Text-to-Speech (TTS) technology or APIs for converting text data back into voice data.

[1153] The "means for playing" refers to a means for allowing the user to listen to the generated audio data through a speaker, earphones, or the like.

[1154] The present invention provides a system for supporting communication for users with hearing limitations. This system converts voice input into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. The following describes an embodiment of the present invention based on specific processing and examples.

[1155] Hardware and software used

[1156] 1. Terminal

[1157] Hardware: Microphone, speaker, communication module (Wi-Fi, 4G / 5G module, etc.)

[1158] Software: Audio capture app, digital signal processing software, communication protocol stack

[1159] 2. Server

[1160] Hardware: High-performance processor, memory, and large storage capacity

[1161] Software: Speech recognition APIs (e.g., Whisper API), text analysis engines, generative AI models (e.g., GPT-3), Text-to-Speech (TTS) APIs

[1162] Processing Description

[1163] 1. Acquiring voice input

[1164] The device uses a microphone to capture the user's voice input as digital audio data, which is then pre-processed with noise reduction and echo cancellation before being stored in a high-quality format and sent directly to the server.

[1165] 2. Sending audio data

[1166] The device then sends the captured high-quality audio data to the server, usually via HTTP requests, but sometimes using proprietary communication protocols.

[1167] 3. Voice Recognition

[1168] The server converts the received voice data into text data using OpenAI's Whisper API, which analyzes the voice data and converts it into a string format, allowing it to accurately transcribe what the user is saying.

[1169] 4. Text Analysis and Response Generation

[1170] The server analyzes the converted text data and generates an appropriate response using a generative AI model (e.g., GPT-3), which understands the user's intent and creates a response that is context-sensitive.

[1171] 5. Speech Synthesis

[1172] The server then sends the generated response text to a Text-to-Speech (TTS) API, which converts the response into audio data, possibly including selecting a voice narrator and setting emotional expressions.

[1173] 6. Sending audio data

[1174] The server then sends the generated audio data back to the device, usually via an HTTP request.

[1175] 7. Audio playback

[1176] The terminal plays the received voice data through the speaker, allowing the user to hear the response from the system.

[1177] Specific examples

[1178] Scenario: A user places an order at a cafe

[1179] Acquiring voice input

[1180] The user speaks to the device, saying, "I'd like a small iced coffee, please." The device captures this speech as high-quality digital audio data.

[1181] Sending audio data

[1182] The device transmits the captured audio data to the server.

[1183] Voice Recognition

[1184] The server uses the Whisper API to convert the received voice data into text, "I'd like a small iced coffee, please."

[1185] Text analysis and response generation

[1186] The server sends a prompt to the generative AI model to generate the appropriate response: "Thank you. Would you like the small size?"

[1187] Speech synthesis

[1188] The server converts this response text into voice data using a TTS API.

[1189] Sending audio data

[1190] The server transmits the generated voice data to the terminal.

[1191] Playing audio

[1192] The terminal plays the received audio data, and the user can hear, "Thank you. Is small size OK?"

[1193] Prompt Sentence Examples

[1194] prompt:

[1195] "If the user says, 'I'd like a small iced coffee, please,' generate an appropriate response."

[1196] As described above, the present invention makes it possible to maintain the quality of digital voice data and generate instantaneous responses, thereby smoothly supporting everyday communication for users with hearing limitations.

[1197] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1198] Step 1:

[1199] The device captures the user's voice input as digital voice data. For example, when the user says, "I'd like a small iced coffee, please," the microphone picks up this voice and converts the analog signal into a digital signal. The input is the user's voice, and the output is digital voice data.

[1200] Step 2:

[1201] The device preprocesses the captured digital audio data. This preprocessing includes noise reduction, echo cancellation, and audio clipping. This results in high-quality audio data. The input is digital audio data, and the output is preprocessed, high-quality digital audio data.

[1202] Step 3:

[1203] The terminal sends the preprocessed audio data to the server. The communication protocol is typically an HTTP POST request. The input is high-quality digital audio data, and the output is a response to the server confirming successful data transmission.

[1204] Step 4:

[1205] The server converts the received voice data into text data using OpenAI's Whisper API. This API analyzes the voice data and converts the user's speech into a string format. The input is high-quality digital voice data, and the output is text data. For example, "I'd like a small iced coffee, please." is converted into text.

[1206] Step 5:

[1207] The server analyzes the text data and generates an appropriate response. A generative AI model (e.g., GPT-3) is used for this analysis and response generation. The input is text data, and the output is an appropriate response text. For example, in response to "I'd like a small iced coffee, please," the server generates the response "Thank you. Is the small size okay?"

[1208] Step 6:

[1209] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts it into voice data. This API converts text data into a voice signal. The input is the response text, and the output is voice data. For example, "Thank you. Is small size okay?" is converted into voice data.

[1210] Step 7:

[1211] The server sends the generated audio data to the device. Again, an HTTP POST request is used. The input is the audio data, and the output is a response indicating successful data transmission to the device.

[1212] Step 8:

[1213] The device then plays the received audio data to the user. At this time, the audio is output using a speaker or earphones. The input is the audio data, and the output is the audio heard by the user. Specifically, the device plays back "Thank you. Is small size OK?"

[1214] (Application example 1)

[1215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1216] People with hearing limitations often face significant limitations in their daily lives, especially when communicating in brick-and-mortar stores. For example, when communicating with store clerks about orders, questions, or confirmations, they may find it difficult to hear what the other person is saying, hindering smooth communication. There is a need for technology that can solve these problems and enable people with hearing limitations to communicate smoothly in brick-and-mortar stores.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1218] In this invention, the server includes means for acquiring a voice input, means for converting the voice input into text data, means for analyzing the text data and generating an appropriate response text, means for converting the response text into voice data, means for playing the voice data, means including an artificial intelligence model for processing the text data and the voice data, means for the system to be executed on a smart device (smartphone, smart glasses, etc.) installed in a physical store, and means for the voice data to facilitate communication in the physical store, thereby enabling people with hearing limitations to smoothly communicate in the physical store, such as when placing an order or asking a question.

[1219] "Means for acquiring voice input" refers to a device or function for capturing voice uttered by a user and inputting it into the system.

[1220] The "means for converting into text data" refers to a device or function for converting acquired voice data into text data.

[1221] The "means for generating an appropriate response text" refers to a device or function for analyzing the converted text data and generating an appropriate response based on the analyzed text data.

[1222] The "means for converting into voice data" refers to a device or function for converting the generated response text into voice data.

[1223] "Means for playing audio data" refers to the function of using a speaker or other playback device to provide the converted audio data to the user.

[1224] "Means including an artificial intelligence model" refers to a device or system function that incorporates artificial intelligence used to analyze text data and generate responses.

[1225] A "smart device" is a portable device, such as a smartphone or smart glasses, that has Internet connectivity and application execution capabilities and is used as a user interface.

[1226] "Means to be installed in a physical store" refers to the means by which a system with functions such as language and voice processing, data transmission and reception, etc. can be operated within a physical store.

[1227] "Means aimed at facilitating communication in physical stores" refers to the functions of devices and systems that improve and support everyday communication such as customer service and transactions in physical stores.

[1228] The present invention provides a system that supports people with hearing impairments to smoothly communicate in physical stores. Specific embodiments are described below in detail.

[1229] System program generation

[1230] This system supports communication through a series of processes: it receives voice input, converts it into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. Specifically, there are three main entities involved in this system: the server, the terminal, and the user.

[1231] Hardware and Software Configuration

[1232] Hardware:

[1233] Portable devices such as smartphones and smart glasses

[1234] Server (cloud-based, e.g. AWS, Google Cloud)

[1235] software:

[1236] Speech Recognition API (OpenAI's Whisper)

[1237] Text analysis and response generation model (OpenAI's GPT-4)

[1238] Speech synthesis API (Google Text-to-Speech)

[1239] Processing flow and data processing

[1240] Getting user voice input

[1241] The device captures the user's voice through the microphone. For example, the user says, "One espresso, please."

[1242] Sending audio data

[1243] The device sends the captured audio data to the server using the HTTP protocol.

[1244] Voice Recognition

[1245] The server uses OpenAI's Whisper API to convert voice data into text. For example, "One espresso please" is converted into text "One espresso please."

[1246] Text analysis and response generation

[1247] The server uses a GPT-4 model to analyze the text data and generate an appropriate response. For example, in response to "One espresso please," the server generates the response "Okay, one espresso please."

[1248] Speech synthesis

[1249] The generated response text is converted into audio data using the Google Text-to-Speech API.

[1250] Sending and playing audio data

[1251] The server sends the generated voice data to the terminal, which plays it back. The user hears the voice saying, "Okay, one espresso, please."

[1252] Specific examples

[1253] This system, which uses a combination of a server and a terminal, is applied as follows:

[1254] Case 1: Ordering at a cafe

[1255] When a user places an order at a cafe, they speak into their smartphone, saying, "One espresso, please." The device recognizes the speech and sends it to the server. The server converts it to text, generates an appropriate response, converts it to audio data, and sends it back to the device. The device then plays back the response, allowing the user to hear it and confirm that their order was received correctly.

[1256] Prompt Sentence Examples

[1257] User: One espresso please.

[1258] AI Response: Ok, one espresso please.

[1259] In this way, the system of the present invention provides a specific embodiment that enables people with hearing limitations to smoothly communicate in physical stores.

[1260] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1261] Step 1:

[1262] Getting user voice input

[1263] Subject: Terminal

[1264] Specific behavior:

[1265] The device captures the user's voice through the built-in microphone.

[1266] Input: User speech (e.g., user says "One espresso please").

[1267] Data processing: The microphone converts analog audio into digital audio data.

[1268] Output: Digital audio data.

[1269] Step 2:

[1270] Sending audio data

[1271] Subject: Terminal

[1272] Specific behavior:

[1273] The device sends the captured audio data to the server.

[1274] Input: Digital audio data.

[1275] Data Calculation: Send data to the server using an HTTP request.

[1276] Output: The audio data sent to the server.

[1277] Step 3:

[1278] Voice Recognition

[1279] Subject: Server

[1280] Specific behavior:

[1281] The server converts the received voice data into text using OpenAI's Whisper API.

[1282] Input: Received audio data.

[1283] Data processing: Voice data is analyzed using the Whisper API and converted into text data.

[1284] Output: Text data (e.g., "One espresso, please").

[1285] Step 4:

[1286] Text analysis and response generation

[1287] Subject: Server

[1288] Specific behavior:

[1289] The server uses the GPT-4 model to analyze the text data and generate an appropriate response.

[1290] Input: Text data ("One espresso please").

[1291] Data computation: Semantic interpretation and response generation using the GPT-4 model.

[1292] Output: The response text data (e.g., "Okay, one espresso, please.").

[1293] Step 5:

[1294] Speech synthesis

[1295] Subject: Server

[1296] Specific behavior:

[1297] The server converts the generated response text into audio data using the Google Text-to-Speech API.

[1298] Input: Response text data.

[1299] Data processing: Convert text data into audio data using the Text-to-Speech API.

[1300] Output: Response voice data (e.g., "Okay, one espresso, please").

[1301] Step 6:

[1302] Sending audio data

[1303] Subject: Server

[1304] Specific behavior:

[1305] The server transmits the generated voice data to the terminal.

[1306] Input: Response audio data.

[1307] Data calculation: The audio data is returned to the terminal as an HTTP response.

[1308] Output: The audio data sent to the device.

[1309] Step 7:

[1310] Playing audio

[1311] Subject: Terminal

[1312] Specific behavior:

[1313] The audio data received by the terminal is played back through the speaker.

[1314] Input: Received audio data.

[1315] Data processing: The speaker converts digital audio data into an analog signal.

[1316] Output: Analogue sound signal (e.g. user hears the voice "Okay, one espresso please").

[1317] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1318] This invention is a system for enabling people with hearing limitations to communicate effectively in everyday life, and is configured with an emotion engine. This system converts voice input into text data, generates an appropriate response, and converts it back into voice data to provide to the user.

[1319] System configuration

[1320] 1. Acquiring voice input (device)

[1321] The device activates the microphone and receives the user's voice input. For example, the user might say, "Please give me a plastic bag at the convenience store." This voice input is captured as digital audio data.

[1322] 2. Sending voice data (terminal → server)

[1323] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[1324] 3. Speech recognition (server)

[1325] The server receives the voice data and converts it into text data using a speech recognition API. For example, the voice saying "Please give me a plastic bag at the convenience store" is converted into text data "Please give me a plastic bag at the convenience store."

[1326] 4. Emotion Recognition (Server)

[1327] The server inputs the text data and voice data into an emotion engine to recognize the user's emotions. For example, it recognizes the user's emotions such as "I'm in a hurry" or "I'm in a hurry" from the tone of the voice and the content of the text.

[1328] 5. Text Analysis and Response Generation (Server)

[1329] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[1330] 6. Speech synthesis (server)

[1331] The server sends the generated response text to a Text-to-Speech (TTS) API and converts it into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[1332] 7. Sending audio data (server → device)

[1333] The server packages the generated audio data and prepares an HTTP response to send to the device.

[1334] 8. Audio playback (device)

[1335] The device decodes the received voice data and plays it through the speaker, and the user can hear the voice saying "It will be ready soon."

[1336] Specific examples

[1337] Scenario: A user places an order at a cafe

[1338] 1. Acquiring voice input (device)

[1339] The user speaks to the terminal, "I'd like a small iced coffee, please."

[1340] The device picks up this audio and captures it as digital audio data.

[1341] 2. Sending voice data (terminal → server)

[1342] The terminal transmits the voice data to the server.

[1343] 3. Speech recognition (server)

[1344] The server converts the speech data into text "I'd like a small iced coffee please."

[1345] 4. Emotion Recognition (Server)

[1346] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[1347] 5. Text Analysis and Response Generation (Server)

[1348] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[1349] 6. Speech synthesis (server)

[1350] The server converts the response text into audio data.

[1351] 7. Sending audio data (server → device)

[1352] The server transmits the audio data to the terminal.

[1353] 8. Audio playback (device)

[1354] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[1355] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate corresponding appropriate responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[1356] The processing flow will be explained below.

[1357] Step 1:

[1358] The device activates the microphone and receives the user's voice input, for example, the user says, "I'd like a coffee, please."

[1359] Step 2:

[1360] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[1361] Step 3:

[1362] The terminal uses an HTTP request to send the audio data to the server.

[1363] Step 4:

[1364] The server analyzes the received voice data and converts it into text data by calling a speech recognition API (e.g., Whisper API). For example, the speech "One coffee please" is converted into the text "One coffee please."

[1365] Step 5:

[1366] The server inputs the text and voice data into an emotion engine to recognize the user's emotions. From the tone of the voice and the content of the text, the server recognizes emotions such as "I'm in a hurry" or "I'm in a hurry."

[1367] Step 6:

[1368] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[1369] Step 7:

[1370] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[1371] Step 8:

[1372] The server packages the generated audio data and prepares an HTTP response to send to the device.

[1373] Step 9:

[1374] The server uses an HTTP response to send the audio data to the terminal.

[1375] Step 10:

[1376] The device decodes the received audio data and plays it through the speaker.

[1377] Step 11:

[1378] The user hears the response, "It will be ready shortly."

[1379] Specific examples

[1380] Scenario: A user places an order at a cafe

[1381] Step 1:

[1382] The user speaks to the terminal, "I'd like a small iced coffee, please."

[1383] Step 2:

[1384] The device picks up this audio and captures it as digital audio data.

[1385] Step 3:

[1386] The terminal transmits the voice data to the server.

[1387] Step 4:

[1388] The server converts the speech data into text "I'd like a small iced coffee please."

[1389] Step 5:

[1390] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[1391] Step 6:

[1392] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[1393] Step 7:

[1394] The server converts the response text into audio data.

[1395] Step 8:

[1396] The server transmits the audio data to the terminal.

[1397] Step 9:

[1398] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[1399] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate appropriate corresponding responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[1400] Example 2

[1401] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1402] Conventional speech recognition systems can only convert speech data into text and generate responses, making it difficult to accurately grasp a user's emotions and intentions. This results in a poor user experience, particularly for people with hearing limitations. Furthermore, because they do not recognize emotions, it is difficult to provide appropriate responses based on the user's situation and feelings. As a result, communication is ineffective, often causing difficulties in daily life.

[1403] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice input into text data, a means for analyzing the text data and voice data and recognizing the user's emotion, and a means for generating an appropriate response text based on the emotion information. This makes it possible to accurately recognize the user's emotion and provide an appropriate response corresponding to that emotion.

[1404] "Audio input" refers to a voice signal emitted by a user into a device, which is captured as digital data.

[1405] "Text data" refers to text information converted from voice input using voice recognition.

[1406] "Audio data" refers to data in which an audio signal is expressed in digital form.

[1407] "Emotion recognition" refers to the process of analyzing speech and text data to identify a user's emotional state.

[1408] "Response text" refers to a textual representation of an appropriate response generated based on the user's utterance and their sentiment.

[1409] "Speech synthesis" refers to the technical process of converting text data into speech data.

[1410] "Playback" refers to audio data being provided to a user in an audible form through an output device such as a speaker.

[1411] "Transmitting and receiving" refers to the process of transferring data from one device to another.

[1412] This invention is a system that enables people with hearing limitations to communicate effectively in their daily lives, and performs a series of processes including voice data acquisition, analysis, emotion recognition, response generation, and voice playback.

[1413] Hardware and software used

[1414] Hardware:

[1415] Device: A device with a microphone and speaker, such as a smartphone or voice assistant device.

[1416] Server: Cloud environment (e.g. AWS, Google Cloud, Azure)

[1417] software:

[1418] Speech Recognition API: Google Cloud Speech-to-Text

[1419] Emotion Recognition Engine: Software that implements a specific emotion recognition algorithm (e.g., a proprietary machine learning model).

[1420] Text-to-Speech (TTS) API: Amazon Polly, etc.

[1421] System Operation Overview

[1422] 1. Acquiring voice input:

[1423] The device activates a microphone and digitally captures what the user says.

[1424] 2. Sending audio data:

[1425] The device sends the captured audio data to the server via an HTTP request.

[1426] 3. Speech Recognition:

[1427] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data.

[1428] 4. Emotion recognition:

[1429] The server inputs the text data obtained by speech recognition and the original speech data into an emotion recognition engine to recognize the user's emotions, such as "tired" or "in a hurry."

[1430] 5. Text analysis and response generation:

[1431] Based on the emotion information, the server generates an appropriate response text.

[1432] 6. Speech synthesis:

[1433] The server sends the generated response text to a Text-to-Speech API such as Amazon Polly, which converts the response into audio data.

[1434] 7. Sending audio data:

[1435] The server sends the generated audio data to the terminal as an HTTP response.

[1436] 8. Audio playback:

[1437] The terminal decodes the received audio data and plays it back to the user through the speaker.

[1438] Specific examples

[1439] Scenario: A user places an order at a cafe

[1440] For example, if a user says to their smartphone, "Please give me a small iced coffee," the system will operate as follows:

[1441] 1. Acquiring voice input:

[1442] The device captures the audio and stores it as digital audio data.

[1443] 2. Sending audio data:

[1444] The terminal transmits the voice data to the server.

[1445] 3. Speech Recognition:

[1446] The server sends the voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "I'd like a small iced coffee, please."

[1447] 4. Emotion recognition:

[1448] The server analyzes the text data and voice data and recognizes the user's emotion as "calm."

[1449] 5. Text analysis and response generation:

[1450] Based on the emotion information, the server generates a response text saying "Thank you. Is small size OK?"

[1451] 6. Speech synthesis:

[1452] The server sends this response text to Amazon Polly, which converts it into voice data.

[1453] 7. Sending audio data:

[1454] The server transmits the generated voice data to the terminal.

[1455] 8. Audio playback:

[1456] The terminal plays the audio data, and the user hears "Thank you. Is small size OK?"

[1457] Prompt Sentence Examples

[1458] Please explain a specific scenario in which a user orders a small iced coffee at a cafe, and provide a detailed explanation of each process: speech recognition, emotion recognition, response generation, speech synthesis, and speech playback.

[1459] The system enables people with hearing limitations to communicate more appropriately and effectively through emotion recognition technology.

[1460] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1461] Step 1: Getting voice input (device)

[1462] Input: User's voice

[1463] Processing: The device activates the microphone and captures the user's voice input as digital audio data.

[1464] Output: Digital audio data

[1465] Specific behavior:

[1466] The device activates the microphone and waits for the user's voice input.

[1467] A user says, "I'd like a small iced coffee at the cafe."

[1468] The device captures this audio digitally and stores it temporarily in memory.

[1469] Step 2: Sending voice data (device → server)

[1470] Input: Digital audio data

[1471] Processing: The device sends the captured audio data to the server via an HTTP POST request, using HTTPS for security reasons.

[1472] Output: Audio data sent to the server

[1473] Specific behavior:

[1474] The audio data is included in the body of the HTTPS request.

[1475] Sends a request to a specific endpoint on the server.

[1476] Step 3: Speech Recognition (Server)

[1477] Input: Audio data sent to the server

[1478] Processing: The server inputs the received audio data into the Google Cloud Speech-to-Text API and converts the audio into text data.

[1479] Output: Text data

[1480] Specific behavior:

[1481] Send the audio data to the Google Cloud Speech-to-Text API.

[1482] Receives the converted text data and passes it to the next process.

[1483] For example, the speech "I'd like a small iced coffee at the cafe, please" is converted into the text "I'd like a small iced coffee at the cafe, please."

[1484] Step 4: Emotion Recognition (Server)

[1485] Input: Text data and audio data

[1486] Processing: The server inputs the text and voice data obtained through speech recognition into an emotion recognition engine to recognize the user's emotions. For example, it may determine that the user is "calm" based on the tone of the voice and the way the words are spoken.

[1487] Output: User's emotional state

[1488] Specific behavior:

[1489] Text data and voice data are input into the emotion recognition engine.

[1490] The emotion information obtained from the emotion recognition engine is passed to the next process.

[1491] For example, the emotion "calm" is identified.

[1492] Step 5: Text analysis and response generation (server)

[1493] Input: Emotion information and text data

[1494] Processing: Based on the emotion information, the server generates an appropriate response text. For example, if the user is recognized as "calm", the response will be "Thank you. Is small size OK?"

[1495] Output: Response text

[1496] Specific behavior:

[1497] The text data and sentiment information are analyzed to determine an appropriate response text.

[1498] Generates response text and passes it to the next process.

[1499] Step 6: Speech synthesis (server)

[1500] Input: Response text

[1501] Processing: The server sends the response text to a Text-to-Speech (TTS) API (e.g., Amazon Polly) and converts it into audio data.

[1502] Output: Audio data

[1503] Specific behavior:

[1504] Send the response text to the TTS API.

[1505] The received audio data is passed to the next process.

[1506] Step 7: Sending audio data (server → device)

[1507] Input: Generated audio data

[1508] Processing: The server sends the generated audio data to the device in an HTTP response. HTTPS is used to ensure security.

[1509] Output: Audio data sent to the device

[1510] Specific behavior:

[1511] The audio data is included in the body of the HTTPS response.

[1512] Sends a response to a specific endpoint on the device.

[1513] Step 8: Playing Audio (Device)

[1514] Input: Audio data sent to the device

[1515] Processing: The device decodes the received audio data and plays it through the speaker. The user can hear the audio "Thank you. Is small size OK?"

[1516] Output: The audio the user can hear

[1517] Specific behavior:

[1518] Decodes audio data and converts it into a playable format.

[1519] Play audio data through a speaker.

[1520] The above is the specific processing flow of this system.

[1521] (Application example 2)

[1522] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1523] It is difficult for people with hearing limitations to communicate effectively in everyday life. In particular, when interacting with complex systems such as self-driving vehicles, it is necessary to generate appropriate responses that take into account the user's emotions. However, current technology has difficulty not only accepting voice input but also accurately recognizing the user's emotions and generating responses based on those emotions, resulting in a decline in the quality of communication.

[1524] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion from voice data and text data, means for generating an appropriate response text taking the emotion into consideration, and means for converting the generated response text into voice data. This makes it possible to recognize the user's emotion and generate an appropriate and prompt response based on the emotion.

[1525] A "means for obtaining voice input" is a device or method for obtaining voice from a user as digital voice data.

[1526] A "means for converting speech input into text data" is a device or method for analyzing acquired speech input and converting it into corresponding text data.

[1527] The "means for recognizing emotions" is a device or method for identifying a user's emotions from text data and voice data.

[1528] The "means for analyzing text data and generating an appropriate response text" is a device or method for analyzing text data in consideration of emotional information and generating an appropriate response.

[1529] The "means for converting response text into voice data" is a device or method for converting the generated response text into voice data.

[1530] The "means for playing audio data" refers to a device or method for playing the generated audio data so that the user can hear it.

[1531] A "means for receiving audio data and converting it into text data for analysis" is a device or method for receiving audio data, analyzing the data, and converting it into text data.

[1532] An "artificial intelligence-based means for generating an appropriate response" is a device or method that uses artificial intelligence techniques to generate an appropriate response based on analytical results and emotional data.

[1533] A "means for converting a generated response into voice data" is a device or method that converts an artificial intelligence-generated text response into voice data.

[1534] The "means for transmitting voice data" is a device or method for transmitting the generated voice data to the terminal.

[1535] "Communication means" refers to a device or method for transmitting and receiving voice data and emotion data between a server and a terminal.

[1536] A "means for generating a response based on analyzed text and emotional information" is a device or method for generating an appropriate response based on the analyzed text and emotional information.

[1537] The present invention provides a system for enabling people with hearing impairments to communicate smoothly in autonomous vehicles. This system includes speech recognition, emotion recognition, response generation, and speech synthesis, which are integrated and operate together.

[1538] System configuration

[1539] 1. Acquiring voice input

[1540] A microphone inside the vehicle captures the user's voice commands and is highly sensitive and has the ability to filter out interior noise.

[1541] 2. Voice Recognition

[1542] The captured voice data is sent from the vehicle's terminal to a server, which then converts the voice data into text using voice recognition software (e.g., Google Speech Recognition).

[1543] 3. Emotion recognition

[1544] The server uses the converted text data and the original voice data to input into an emotion recognition engine, which identifies the user's emotions from the tone of the voice and keywords used. This engine implements natural language processing and machine learning algorithms.

[1545] 4. Response Generation

[1546] The server generates an appropriate response based on the emotional information obtained by the emotion recognition engine. A generative AI model is used to generate the response. This AI model considers the emotional information and the content of the text data to propose the optimal response text.

[1547] 5. Speech Synthesis

[1548] The response text is converted to voice data on the server using Text-to-Speech (TTS) technology (e.g., pyttsx3), which mimics human voice tones and communicates to the user with natural pronunciation.

[1549] 6. Audio playback

[1550] The generated audio data is played back through the vehicle's speakers, allowing the user to hear the system's response.

[1551] Example of a system

[1552] For example, if a user in a self-driving vehicle says, "Turn left at the next intersection," the system will:

[1553] 1. Acquiring voice input

[1554] The user's instruction, "Turn left at the next intersection," is picked up by a microphone inside the car.

[1555] 2. Voice Recognition

[1556] The in-car device sends the voice data to the server, and Google Speech Recognition converts this data into text such as "Turn left at the next intersection."

[1557] 3. Emotion recognition

[1558] The server uses this text data and voice data to have an emotion recognition engine determine that the user is "confused."

[1559] 4. Response Generation

[1560] The server uses a generative AI model to generate a response based on emotion and text: "Okay, turn left at the next intersection."

[1561] 5. Speech Synthesis

[1562] The server uses text-to-speech technology to convert the generated response text into voice data.

[1563] 6. Audio playback

[1564] The response "Okay, turn left at the next intersection" is played over the car's speakers.

[1565] Prompt Sentence Examples

[1566] 001: "Turn left at the next intersection."

[1567] 002: System interprets the user's tone as confused and gives the response: "Okay. Turn left at the next intersection."

[1568] The system can accurately recognize a user's emotions and generate quick and appropriate responses based on those emotions, significantly improving communication for hearing-impaired users.

[1569] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1570] Step 1:

[1571] The user provides voice input, for example, "Turn left at the next intersection." The voice input is captured as digital voice data by a microphone inside the vehicle.

[1572] Step 2:

[1573] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol, and the server receives the data.

[1574] Step 3:

[1575] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech Recognition). For example, it generates text data such as "Turn left at the next intersection."

[1576] Step 4:

[1577] The server receives the text and voice data and inputs it into an emotion recognition engine, which uses natural language processing and machine learning algorithms to identify the user's emotion. For example, it outputs emotional information such as "confused."

[1578] Step 5:

[1579] The server uses a generative AI model based on emotion information and text data to generate an appropriate response. For example, in response to the emotion "confusion," it generates the response text "I understand. Turn left at the next intersection."

[1580] Step 6:

[1581] The server converts the generated response text into voice data using Text-to-Speech (TTS) technology (e.g., pyttsx3). For example, it generates voice data that says, "Okay, turn left at the next intersection."

[1582] Step 7:

[1583] The server sends the generated audio data to the device using an HTTP response, and the device receives the data.

[1584] Step 8:

[1585] The device then plays the received voice data through the car's speakers, allowing the user to hear the voice saying, "Okay, turn left at the next intersection."

[1586] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1587] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1588] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1589] [Fourth embodiment]

[1590] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1591] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1592] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1593] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1594] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1595] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1596] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1597] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1598] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1599] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1600] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1601] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1602] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1603] The present invention is a system for assisting people with hearing limitations in their daily communication by converting speech input into text data, generating appropriate responses, and converting them back into speech data for presentation to the user.

[1604] System configuration

[1605] 1. Acquiring voice input (device)

[1606] The device uses a microphone to receive user voice input, which is captured as digital audio data.

[1607] For example, a user might say, "Please give me a plastic bag at the convenience store."

[1608] 2. Sending voice data (terminal → server)

[1609] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[1610] 3. Speech recognition (server)

[1611] The server processes the received voice data and converts it into text data using OpenAI's Whisper API.

[1612] For example, speech data uttered "Please give me a plastic bag at the convenience store" is converted into text data stating "Please give me a plastic bag at the convenience store."

[1613] 4. Text analysis and response generation (server)

[1614] The server then uses artificial intelligence to analyze the converted text data, which allows it to understand the meaning of the text and generate an appropriate response.

[1615] For example, in response to the text data "Please give me a plastic bag at the convenience store," the AI ​​generates the response "Yes, would you like a plastic bag?"

[1616] 5. Speech synthesis (server)

[1617] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data.

[1618] The response text "Yes, would you like a plastic bag?" is converted into speech data.

[1619] 6. Sending audio data (server → device)

[1620] The server transmits the generated voice data to the terminal, which receives it and provides a response to the user.

[1621] 7. Audio playback (device)

[1622] The device then plays the received voice data through the speaker, and the user hears the voice saying, "Yes, would you like a plastic bag?"

[1623] Specific examples

[1624] Scenario: A user places an order at a cafe

[1625] 1. Acquiring voice input (device)

[1626] The user speaks to the terminal, "I'd like a small iced coffee, please."

[1627] The device picks up this audio and captures it as digital audio data.

[1628] 2. Sending voice data (terminal → server)

[1629] The terminal transmits the voice data to the server.

[1630] 3. Speech recognition (server)

[1631] The server converts the speech data into text "I'd like a small iced coffee please."

[1632] 4. Text analysis and response generation (server)

[1633] The server uses artificial intelligence to analyze the text data and generate an appropriate response: "Thank you. Is small size okay?"

[1634] 5. Speech synthesis (server)

[1635] The server converts the response text into audio data.

[1636] 6. Sending audio data (server → device)

[1637] The server transmits the audio data to the terminal.

[1638] 7. Audio playback (device)

[1639] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[1640] In this way, the present invention supports everyday communication for people with hearing limitations through a series of processes that convert voice data into text data, generate an appropriate response, and provide it to the user again as voice data.

[1641] The processing flow will be explained below.

[1642] Step 1:

[1643] The device activates the microphone and receives voice input, for example, when the user says, "I'd like a coffee, please."

[1644] Step 2:

[1645] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[1646] Step 3:

[1647] The terminal uses an HTTP request to send the audio data to the server.

[1648] Step 4:

[1649] The server analyzes the received voice data and converts it into text data by calling a voice recognition API (e.g. Whisper API). For example, the voice saying "One coffee please" is converted into the text "One coffee please."

[1650] Step 5:

[1651] The server sends the text data to an artificial intelligence system that analyzes it to generate an appropriate response, such as "Hot or Ice?"

[1652] Step 6:

[1653] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into speech data. For example, the text "Is it hot or iced?" is converted into speech data.

[1654] Step 7:

[1655] The server packages the generated audio data and prepares an HTTP response to send to the device.

[1656] Step 8:

[1657] The server uses an HTTP response to send the audio data to the terminal.

[1658] Step 9:

[1659] The device decodes the received audio data and plays it through the speaker.

[1660] Step 10:

[1661] The user hears a voice response saying, "Hot or iced?"

[1662] The above is the specific program processing flow of the system, which allows users with hearing limitations to communicate smoothly.

[1663] Example 1

[1664] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1665] It is difficult for people with hearing limitations to communicate smoothly in their daily lives. Current communication support systems convert voice input into text, generate an appropriate response, and then convert it back into voice to provide to the user. However, these systems often lack the quality, immediacy, and appropriate response generation of the voice data. Processing delays and unstable communication are also problems. A system that can solve these issues and provide fast, high-quality communication support is needed.

[1666] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1667] In this invention, the server includes a preprocessing means for converting a voice input into digital voice data and generating high-quality voice data, a communication means for transmitting the high-quality voice data to an analysis server, a means for converting the high-quality voice data into text data, a means for analyzing the text data and generating an appropriate response text, a means for converting the response text into voice data, and a means for receiving and playing the voice data, thereby enabling the generation of high-quality voice data and instantaneous response generation.

[1668] "Audio input" refers to words or sounds made by a user through a microphone or other sound capturing device.

[1669] "Digital audio data" means data that has been converted from analog audio into a digital signal for further processing or communication.

[1670] The "pre-processing means" refers to a means for performing processes such as noise removal and echo cancellation when converting captured audio into high-quality digital audio data.

[1671] "Communication means" refers to the means for sending and receiving data between the terminal and the server. Typically, the HTTP protocol or a dedicated protocol is used via the Internet.

[1672] "Text data" refers to data in the form of a string of characters obtained by analyzing voice data. This data is in a format that is easy for computers to process.

[1673] "Analysis means" refers to artificial intelligence or algorithms that understand the content of text data and generate appropriate responses.

[1674] "Appropriate response text" refers to text data in response to text data from a user that is generated by an AI model, etc.

[1675] "Means for converting into voice data" refers to Text-to-Speech (TTS) technology or APIs for converting text data back into voice data.

[1676] The "means for playing" refers to a means for allowing the user to listen to the generated audio data through a speaker, earphones, or the like.

[1677] The present invention provides a system for supporting communication for users with hearing limitations. This system converts voice input into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. The following describes an embodiment of the present invention based on specific processing and examples.

[1678] Hardware and software used

[1679] 1. Terminal

[1680] Hardware: Microphone, speaker, communication module (Wi-Fi, 4G / 5G module, etc.)

[1681] Software: Audio capture app, digital signal processing software, communication protocol stack

[1682] 2. Server

[1683] Hardware: High-performance processor, memory, and large storage capacity

[1684] Software: Speech recognition APIs (e.g., Whisper API), text analysis engines, generative AI models (e.g., GPT-3), Text-to-Speech (TTS) APIs

[1685] Processing Description

[1686] 1. Acquiring voice input

[1687] The device uses a microphone to capture the user's voice input as digital audio data, which is then pre-processed with noise reduction and echo cancellation before being stored in a high-quality format and sent directly to the server.

[1688] 2. Sending audio data

[1689] The device then sends the captured high-quality audio data to the server, usually via HTTP requests, but sometimes using proprietary communication protocols.

[1690] 3. Voice Recognition

[1691] The server converts the received voice data into text data using OpenAI's Whisper API, which analyzes the voice data and converts it into a string format, allowing it to accurately transcribe what the user is saying.

[1692] 4. Text Analysis and Response Generation

[1693] The server analyzes the converted text data and generates an appropriate response using a generative AI model (e.g., GPT-3), which understands the user's intent and creates a response that is context-sensitive.

[1694] 5. Speech Synthesis

[1695] The server then sends the generated response text to a Text-to-Speech (TTS) API, which converts the response into audio data, possibly including selecting a voice narrator and setting emotional expressions.

[1696] 6. Sending audio data

[1697] The server then sends the generated audio data back to the device, usually via an HTTP request.

[1698] 7. Audio playback

[1699] The terminal plays the received voice data through the speaker, allowing the user to hear the response from the system.

[1700] Specific examples

[1701] Scenario: A user places an order at a cafe

[1702] Acquiring voice input

[1703] The user speaks to the device, saying, "I'd like a small iced coffee, please." The device captures this speech as high-quality digital audio data.

[1704] Sending audio data

[1705] The device transmits the captured audio data to the server.

[1706] Voice Recognition

[1707] The server uses the Whisper API to convert the received voice data into text, "I'd like a small iced coffee, please."

[1708] Text analysis and response generation

[1709] The server sends a prompt to the generative AI model to generate the appropriate response: "Thank you. Would you like the small size?"

[1710] Speech synthesis

[1711] The server converts this response text into voice data using a TTS API.

[1712] Sending audio data

[1713] The server transmits the generated voice data to the terminal.

[1714] Playing audio

[1715] The terminal plays the received audio data, and the user can hear, "Thank you. Is small size OK?"

[1716] Prompt Sentence Examples

[1717] prompt:

[1718] "If the user says, 'I'd like a small iced coffee, please,' generate an appropriate response."

[1719] As described above, the present invention makes it possible to maintain the quality of digital voice data and generate instantaneous responses, thereby smoothly supporting everyday communication for users with hearing limitations.

[1720] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1721] Step 1:

[1722] The device captures the user's voice input as digital voice data. For example, when the user says, "I'd like a small iced coffee, please," the microphone picks up this voice and converts the analog signal into a digital signal. The input is the user's voice, and the output is digital voice data.

[1723] Step 2:

[1724] The device preprocesses the captured digital audio data. This preprocessing includes noise reduction, echo cancellation, and audio clipping. This results in high-quality audio data. The input is digital audio data, and the output is preprocessed, high-quality digital audio data.

[1725] Step 3:

[1726] The terminal sends the preprocessed audio data to the server. The communication protocol is typically an HTTP POST request. The input is high-quality digital audio data, and the output is a response to the server confirming successful data transmission.

[1727] Step 4:

[1728] The server converts the received voice data into text data using OpenAI's Whisper API. This API analyzes the voice data and converts the user's speech into a string format. The input is high-quality digital voice data, and the output is text data. For example, "I'd like a small iced coffee, please." is converted into text.

[1729] Step 5:

[1730] The server analyzes the text data and generates an appropriate response. A generative AI model (e.g., GPT-3) is used for this analysis and response generation. The input is text data, and the output is an appropriate response text. For example, in response to "I'd like a small iced coffee, please," the server generates the response "Thank you. Is the small size okay?"

[1731] Step 6:

[1732] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts it into voice data. This API converts text data into a voice signal. The input is the response text, and the output is voice data. For example, "Thank you. Is small size okay?" is converted into voice data.

[1733] Step 7:

[1734] The server sends the generated audio data to the device. Again, an HTTP POST request is used. The input is the audio data, and the output is a response indicating successful data transmission to the device.

[1735] Step 8:

[1736] The device then plays the received audio data to the user. At this time, the audio is output using a speaker or earphones. The input is the audio data, and the output is the audio heard by the user. Specifically, the device plays back "Thank you. Is small size OK?"

[1737] (Application example 1)

[1738] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1739] People with hearing limitations often face significant limitations in their daily lives, especially when communicating in brick-and-mortar stores. For example, when communicating with store clerks about orders, questions, or confirmations, they may find it difficult to hear what the other person is saying, hindering smooth communication. There is a need for technology that can solve these problems and enable people with hearing limitations to communicate smoothly in brick-and-mortar stores.

[1740] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1741] In this invention, the server includes means for acquiring a voice input, means for converting the voice input into text data, means for analyzing the text data and generating an appropriate response text, means for converting the response text into voice data, means for playing the voice data, means including an artificial intelligence model for processing the text data and the voice data, means for the system to be executed on a smart device (smartphone, smart glasses, etc.) installed in a physical store, and means for the voice data to facilitate communication in the physical store, thereby enabling people with hearing limitations to smoothly communicate in the physical store, such as when placing an order or asking a question.

[1742] "Means for acquiring voice input" refers to a device or function for capturing voice uttered by a user and inputting it into the system.

[1743] The "means for converting into text data" refers to a device or function for converting acquired voice data into text data.

[1744] The "means for generating an appropriate response text" refers to a device or function for analyzing the converted text data and generating an appropriate response based on the analyzed text data.

[1745] The "means for converting into voice data" refers to a device or function for converting the generated response text into voice data.

[1746] "Means for playing audio data" refers to the function of using a speaker or other playback device to provide the converted audio data to the user.

[1747] "Means including an artificial intelligence model" refers to a device or system function that incorporates artificial intelligence used to analyze text data and generate responses.

[1748] A "smart device" is a portable device, such as a smartphone or smart glasses, that has Internet connectivity and application execution capabilities and is used as a user interface.

[1749] "Means to be installed in a physical store" refers to the means by which a system with functions such as language and voice processing, data transmission and reception, etc. can be operated within a physical store.

[1750] "Means aimed at facilitating communication in physical stores" refers to the functions of devices and systems that improve and support everyday communication such as customer service and transactions in physical stores.

[1751] The present invention provides a system that supports people with hearing impairments to smoothly communicate in physical stores. Specific embodiments are described below in detail.

[1752] System program generation

[1753] This system supports communication through a series of processes: it receives voice input, converts it into text data, generates an appropriate response, and then converts it back into voice data to provide to the user. Specifically, there are three main entities involved in this system: the server, the terminal, and the user.

[1754] Hardware and Software Configuration

[1755] Hardware:

[1756] Portable devices such as smartphones and smart glasses

[1757] Server (cloud-based, e.g. AWS, Google Cloud)

[1758] software:

[1759] Speech Recognition API (OpenAI's Whisper)

[1760] Text analysis and response generation model (OpenAI's GPT-4)

[1761] Speech synthesis API (Google Text-to-Speech)

[1762] Processing flow and data processing

[1763] Getting user voice input

[1764] The device captures the user's voice through the microphone. For example, the user says, "One espresso, please."

[1765] Sending audio data

[1766] The device sends the captured audio data to the server using the HTTP protocol.

[1767] Voice Recognition

[1768] The server uses OpenAI's Whisper API to convert voice data into text. For example, "One espresso please" is converted into text "One espresso please."

[1769] Text analysis and response generation

[1770] The server uses a GPT-4 model to analyze the text data and generate an appropriate response. For example, in response to "One espresso please," the server generates the response "Okay, one espresso please."

[1771] Speech synthesis

[1772] The generated response text is converted into audio data using the Google Text-to-Speech API.

[1773] Sending and playing audio data

[1774] The server sends the generated voice data to the terminal, which plays it back. The user hears the voice saying, "Okay, one espresso, please."

[1775] Specific examples

[1776] This system, which uses a combination of a server and a terminal, is applied as follows:

[1777] Case 1: Ordering at a cafe

[1778] When a user places an order at a cafe, they speak into their smartphone, saying, "One espresso, please." The device recognizes the speech and sends it to the server. The server converts it to text, generates an appropriate response, converts it to audio data, and sends it back to the device. The device then plays back the response, allowing the user to hear it and confirm that their order was received correctly.

[1779] Prompt Sentence Examples

[1780] User: One espresso please.

[1781] AI Response: Ok, one espresso please.

[1782] In this way, the system of the present invention provides a specific embodiment that enables people with hearing limitations to smoothly communicate in physical stores.

[1783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1784] Step 1:

[1785] Getting user voice input

[1786] Subject: Terminal

[1787] Specific behavior:

[1788] The device captures the user's voice through the built-in microphone.

[1789] Input: User speech (e.g., user says "One espresso please").

[1790] Data processing: The microphone converts analog audio into digital audio data.

[1791] Output: Digital audio data.

[1792] Step 2:

[1793] Sending audio data

[1794] Subject: Terminal

[1795] Specific behavior:

[1796] The device sends the captured audio data to the server.

[1797] Input: Digital audio data.

[1798] Data Calculation: Send data to the server using an HTTP request.

[1799] Output: The audio data sent to the server.

[1800] Step 3:

[1801] Voice Recognition

[1802] Subject: Server

[1803] Specific behavior:

[1804] The server converts the received voice data into text using OpenAI's Whisper API.

[1805] Input: Received audio data.

[1806] Data processing: Voice data is analyzed using the Whisper API and converted into text data.

[1807] Output: Text data (e.g., "One espresso, please").

[1808] Step 4:

[1809] Text analysis and response generation

[1810] Subject: Server

[1811] Specific behavior:

[1812] The server uses the GPT-4 model to analyze the text data and generate an appropriate response.

[1813] Input: Text data ("One espresso please").

[1814] Data computation: Semantic interpretation and response generation using the GPT-4 model.

[1815] Output: The response text data (e.g., "Okay, one espresso, please.").

[1816] Step 5:

[1817] Speech synthesis

[1818] Subject: Server

[1819] Specific behavior:

[1820] The server converts the generated response text into audio data using the Google Text-to-Speech API.

[1821] Input: Response text data.

[1822] Data processing: Convert text data into audio data using the Text-to-Speech API.

[1823] Output: Response voice data (e.g., "Okay, one espresso, please").

[1824] Step 6:

[1825] Sending audio data

[1826] Subject: Server

[1827] Specific behavior:

[1828] The server transmits the generated voice data to the terminal.

[1829] Input: Response audio data.

[1830] Data calculation: The audio data is returned to the terminal as an HTTP response.

[1831] Output: The audio data sent to the device.

[1832] Step 7:

[1833] Playing audio

[1834] Subject: Terminal

[1835] Specific behavior:

[1836] The audio data received by the terminal is played back through the speaker.

[1837] Input: Received audio data.

[1838] Data processing: The speaker converts digital audio data into an analog signal.

[1839] Output: Analogue sound signal (e.g. user hears the voice "Okay, one espresso please").

[1840] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1841] This invention is a system for enabling people with hearing limitations to communicate effectively in everyday life, and is configured with an emotion engine. This system converts voice input into text data, generates an appropriate response, and converts it back into voice data to provide to the user.

[1842] System configuration

[1843] 1. Acquiring voice input (device)

[1844] The device activates the microphone and receives the user's voice input. For example, the user might say, "Please give me a plastic bag at the convenience store." This voice input is captured as digital audio data.

[1845] 2. Sending voice data (terminal → server)

[1846] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol.

[1847] 3. Speech recognition (server)

[1848] The server receives the voice data and converts it into text data using a speech recognition API. For example, the voice saying "Please give me a plastic bag at the convenience store" is converted into text data "Please give me a plastic bag at the convenience store."

[1849] 4. Emotion Recognition (Server)

[1850] The server inputs the text data and voice data into an emotion engine to recognize the user's emotions. For example, it recognizes the user's emotions such as "I'm in a hurry" or "I'm in a hurry" from the tone of the voice and the content of the text.

[1851] 5. Text Analysis and Response Generation (Server)

[1852] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[1853] 6. Speech synthesis (server)

[1854] The server sends the generated response text to a Text-to-Speech (TTS) API and converts it into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[1855] 7. Sending audio data (server → device)

[1856] The server packages the generated audio data and prepares an HTTP response to send to the device.

[1857] 8. Audio playback (device)

[1858] The device decodes the received voice data and plays it through the speaker, and the user can hear the voice saying "It will be ready soon."

[1859] Specific examples

[1860] Scenario: A user places an order at a cafe

[1861] 1. Acquiring voice input (device)

[1862] The user speaks to the terminal, "I'd like a small iced coffee, please."

[1863] The device picks up this audio and captures it as digital audio data.

[1864] 2. Sending voice data (terminal → server)

[1865] The terminal transmits the voice data to the server.

[1866] 3. Speech recognition (server)

[1867] The server converts the speech data into text "I'd like a small iced coffee please."

[1868] 4. Emotion Recognition (Server)

[1869] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[1870] 5. Text Analysis and Response Generation (Server)

[1871] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[1872] 6. Speech synthesis (server)

[1873] The server converts the response text into audio data.

[1874] 7. Sending audio data (server → device)

[1875] The server transmits the audio data to the terminal.

[1876] 8. Audio playback (device)

[1877] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[1878] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate corresponding appropriate responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[1879] The processing flow will be explained below.

[1880] Step 1:

[1881] The device activates the microphone and receives the user's voice input, for example, the user says, "I'd like a coffee, please."

[1882] Step 2:

[1883] The device converts the captured audio data into a digital format and prepares an HTTP request to send to the server.

[1884] Step 3:

[1885] The terminal uses an HTTP request to send the audio data to the server.

[1886] Step 4:

[1887] The server analyzes the received voice data and converts it into text data by calling a speech recognition API (e.g., Whisper API). For example, the speech "One coffee please" is converted into the text "One coffee please."

[1888] Step 5:

[1889] The server inputs the text and voice data into an emotion engine to recognize the user's emotions. From the tone of the voice and the content of the text, the server recognizes emotions such as "I'm in a hurry" or "I'm in a hurry."

[1890] Step 6:

[1891] The server analyzes the text data using the emotional information recognized by the emotion engine and generates an appropriate response. For example, if the user is recognized as "in a hurry," the response will be "We'll have it ready right away."

[1892] Step 7:

[1893] The server sends the generated response text to a Text-to-Speech (TTS) API, which converts the text data into voice data. For example, the response text "We'll have it ready soon" is converted into voice data.

[1894] Step 8:

[1895] The server packages the generated audio data and prepares an HTTP response to send to the device.

[1896] Step 9:

[1897] The server uses an HTTP response to send the audio data to the terminal.

[1898] Step 10:

[1899] The device decodes the received audio data and plays it through the speaker.

[1900] Step 11:

[1901] The user hears the response, "It will be ready shortly."

[1902] Specific examples

[1903] Scenario: A user places an order at a cafe

[1904] Step 1:

[1905] The user speaks to the terminal, "I'd like a small iced coffee, please."

[1906] Step 2:

[1907] The device picks up this audio and captures it as digital audio data.

[1908] Step 3:

[1909] The terminal transmits the voice data to the server.

[1910] Step 4:

[1911] The server converts the speech data into text "I'd like a small iced coffee please."

[1912] Step 5:

[1913] The server inputs the text and voice data into an emotion engine to recognize the user's emotion. For example, it can determine that the user is "calm" based on their tone and choice of words.

[1914] Step 6:

[1915] The server analyzes the text data using the emotional information recognized by the emotion engine and generates a response: "Thank you. Is small size okay?"

[1916] Step 7:

[1917] The server converts the response text into audio data.

[1918] Step 8:

[1919] The server transmits the audio data to the terminal.

[1920] Step 9:

[1921] The terminal plays the audio data and the user asks, "Thank you. Is small size OK?"

[1922] In this way, the present invention uses an emotion engine to recognize a user's emotions and generate appropriate corresponding responses, thereby providing more sophisticated and effective support for everyday communication for people with hearing limitations.

[1923] Example 2

[1924] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1925] Conventional speech recognition systems can only convert speech data into text and generate responses, making it difficult to accurately grasp a user's emotions and intentions. This results in a poor user experience, particularly for people with hearing limitations. Furthermore, because they do not recognize emotions, it is difficult to provide appropriate responses based on the user's situation and feelings. As a result, communication is ineffective, often causing difficulties in daily life.

[1926] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice input into text data, a means for analyzing the text data and voice data and recognizing the user's emotion, and a means for generating an appropriate response text based on the emotion information. This makes it possible to accurately recognize the user's emotion and provide an appropriate response corresponding to that emotion.

[1927] "Audio input" refers to a voice signal emitted by a user into a device, which is captured as digital data.

[1928] "Text data" refers to text information converted from voice input using voice recognition.

[1929] "Audio data" refers to data in which an audio signal is expressed in digital form.

[1930] "Emotion recognition" refers to the process of analyzing speech and text data to identify a user's emotional state.

[1931] "Response text" refers to a textual representation of an appropriate response generated based on the user's utterance and their sentiment.

[1932] "Speech synthesis" refers to the technical process of converting text data into speech data.

[1933] "Playback" refers to audio data being provided to a user in an audible form through an output device such as a speaker.

[1934] "Transmitting and receiving" refers to the process of transferring data from one device to another.

[1935] This invention is a system that enables people with hearing limitations to communicate effectively in their daily lives, and performs a series of processes including voice data acquisition, analysis, emotion recognition, response generation, and voice playback.

[1936] Hardware and software used

[1937] Hardware:

[1938] Device: A device with a microphone and speaker, such as a smartphone or voice assistant device.

[1939] Server: Cloud environment (e.g. AWS, Google Cloud, Azure)

[1940] software:

[1941] Speech Recognition API: Google Cloud Speech-to-Text

[1942] Emotion Recognition Engine: Software that implements a specific emotion recognition algorithm (e.g., a proprietary machine learning model).

[1943] Text-to-Speech (TTS) API: Amazon Polly, etc.

[1944] System Operation Overview

[1945] 1. Acquiring voice input:

[1946] The device activates a microphone and digitally captures what the user says.

[1947] 2. Sending audio data:

[1948] The device sends the captured audio data to the server via an HTTP request.

[1949] 3. Speech Recognition:

[1950] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data.

[1951] 4. Emotion recognition:

[1952] The server inputs the text data obtained by speech recognition and the original speech data into an emotion recognition engine to recognize the user's emotions, such as "tired" or "in a hurry."

[1953] 5. Text analysis and response generation:

[1954] Based on the emotion information, the server generates an appropriate response text.

[1955] 6. Speech synthesis:

[1956] The server sends the generated response text to a Text-to-Speech API such as Amazon Polly, which converts the response into audio data.

[1957] 7. Sending audio data:

[1958] The server sends the generated audio data to the terminal as an HTTP response.

[1959] 8. Audio playback:

[1960] The terminal decodes the received audio data and plays it back to the user through the speaker.

[1961] Specific examples

[1962] Scenario: A user places an order at a cafe

[1963] For example, if a user says to their smartphone, "Please give me a small iced coffee," the system will operate as follows:

[1964] 1. Acquiring voice input:

[1965] The device captures the audio and stores it as digital audio data.

[1966] 2. Sending audio data:

[1967] The terminal transmits the voice data to the server.

[1968] 3. Speech Recognition:

[1969] The server sends the voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "I'd like a small iced coffee, please."

[1970] 4. Emotion recognition:

[1971] The server analyzes the text data and voice data and recognizes the user's emotion as "calm."

[1972] 5. Text analysis and response generation:

[1973] Based on the emotion information, the server generates a response text saying "Thank you. Is small size OK?"

[1974] 6. Speech synthesis:

[1975] The server sends this response text to Amazon Polly, which converts it into voice data.

[1976] 7. Sending audio data:

[1977] The server transmits the generated voice data to the terminal.

[1978] 8. Audio playback:

[1979] The terminal plays the audio data, and the user hears "Thank you. Is small size OK?"

[1980] Prompt Sentence Examples

[1981] Please explain a specific scenario in which a user orders a small iced coffee at a cafe, and provide a detailed explanation of each process: speech recognition, emotion recognition, response generation, speech synthesis, and speech playback.

[1982] The system enables people with hearing limitations to communicate more appropriately and effectively through emotion recognition technology.

[1983] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1984] Step 1: Getting voice input (device)

[1985] Input: User's voice

[1986] Processing: The device activates the microphone and captures the user's voice input as digital audio data.

[1987] Output: Digital audio data

[1988] Specific behavior:

[1989] The device activates the microphone and waits for the user's voice input.

[1990] A user says, "I'd like a small iced coffee at the cafe."

[1991] The device captures this audio digitally and stores it temporarily in memory.

[1992] Step 2: Sending voice data (device → server)

[1993] Input: Digital audio data

[1994] Processing: The device sends the captured audio data to the server via an HTTP POST request, using HTTPS for security reasons.

[1995] Output: Audio data sent to the server

[1996] Specific behavior:

[1997] The audio data is included in the body of the HTTPS request.

[1998] Sends a request to a specific endpoint on the server.

[1999] Step 3: Speech Recognition (Server)

[2000] Input: Audio data sent to the server

[2001] Processing: The server inputs the received audio data into the Google Cloud Speech-to-Text API and converts the audio into text data.

[2002] Output: Text data

[2003] Specific behavior:

[2004] Send the audio data to the Google Cloud Speech-to-Text API.

[2005] Receives the converted text data and passes it to the next process.

[2006] For example, the speech "I'd like a small iced coffee at the cafe, please" is converted into the text "I'd like a small iced coffee at the cafe, please."

[2007] Step 4: Emotion Recognition (Server)

[2008] Input: Text data and audio data

[2009] Processing: The server inputs the text and voice data obtained through speech recognition into an emotion recognition engine to recognize the user's emotions. For example, it may determine that the user is "calm" based on the tone of the voice and the way the words are spoken.

[2010] Output: User's emotional state

[2011] Specific behavior:

[2012] Text data and voice data are input into the emotion recognition engine.

[2013] The emotion information obtained from the emotion recognition engine is passed to the next process.

[2014] For example, the emotion "calm" is identified.

[2015] Step 5: Text analysis and response generation (server)

[2016] Input: Emotion information and text data

[2017] Processing: Based on the emotion information, the server generates an appropriate response text. For example, if the user is recognized as "calm", the response will be "Thank you. Is small size OK?"

[2018] Output: Response text

[2019] Specific behavior:

[2020] The text data and sentiment information are analyzed to determine an appropriate response text.

[2021] Generates response text and passes it to the next process.

[2022] Step 6: Speech synthesis (server)

[2023] Input: Response text

[2024] Processing: The server sends the response text to a Text-to-Speech (TTS) API (e.g., Amazon Polly) and converts it into audio data.

[2025] Output: Audio data

[2026] Specific behavior:

[2027] Send the response text to the TTS API.

[2028] The received audio data is passed to the next process.

[2029] Step 7: Sending audio data (server → device)

[2030] Input: Generated audio data

[2031] Processing: The server sends the generated audio data to the device in an HTTP response. HTTPS is used to ensure security.

[2032] Output: Audio data sent to the device

[2033] Specific behavior:

[2034] The audio data is included in the body of the HTTPS response.

[2035] Sends a response to a specific endpoint on the device.

[2036] Step 8: Playing Audio (Device)

[2037] Input: Audio data sent to the device

[2038] Processing: The device decodes the received audio data and plays it through the speaker. The user can hear the audio "Thank you. Is small size OK?"

[2039] Output: The audio the user can hear

[2040] Specific behavior:

[2041] Decodes audio data and converts it into a playable format.

[2042] Play audio data through a speaker.

[2043] The above is the specific processing flow of this system.

[2044] (Application example 2)

[2045] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2046] It is difficult for people with hearing limitations to communicate effectively in everyday life. In particular, when interacting with complex systems such as self-driving vehicles, it is necessary to generate appropriate responses that take into account the user's emotions. However, current technology has difficulty not only accepting voice input but also accurately recognizing the user's emotions and generating responses based on those emotions, resulting in a decline in the quality of communication.

[2047] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing a user's emotion from voice data and text data, means for generating an appropriate response text taking the emotion into consideration, and means for converting the generated response text into voice data. This makes it possible to recognize the user's emotion and generate an appropriate and prompt response based on the emotion.

[2048] A "means for obtaining voice input" is a device or method for obtaining voice from a user as digital voice data.

[2049] A "means for converting speech input into text data" is a device or method for analyzing acquired speech input and converting it into corresponding text data.

[2050] The "means for recognizing emotions" is a device or method for identifying a user's emotions from text data and voice data.

[2051] The "means for analyzing text data and generating an appropriate response text" is a device or method for analyzing text data in consideration of emotional information and generating an appropriate response.

[2052] The "means for converting response text into voice data" is a device or method for converting the generated response text into voice data.

[2053] The "means for playing audio data" refers to a device or method for playing the generated audio data so that the user can hear it.

[2054] A "means for receiving audio data and converting it into text data for analysis" is a device or method for receiving audio data, analyzing the data, and converting it into text data.

[2055] An "artificial intelligence-based means for generating an appropriate response" is a device or method that uses artificial intelligence techniques to generate an appropriate response based on analytical results and emotional data.

[2056] A "means for converting a generated response into voice data" is a device or method that converts an artificial intelligence-generated text response into voice data.

[2057] The "means for transmitting voice data" is a device or method for transmitting the generated voice data to the terminal.

[2058] "Communication means" refers to a device or method for transmitting and receiving voice data and emotion data between a server and a terminal.

[2059] A "means for generating a response based on analyzed text and emotional information" is a device or method for generating an appropriate response based on the analyzed text and emotional information.

[2060] The present invention provides a system for enabling people with hearing impairments to communicate smoothly in autonomous vehicles. This system includes speech recognition, emotion recognition, response generation, and speech synthesis, which are integrated and operate together.

[2061] System configuration

[2062] 1. Acquiring voice input

[2063] A microphone inside the vehicle captures the user's voice commands and is highly sensitive and has the ability to filter out interior noise.

[2064] 2. Voice Recognition

[2065] The captured voice data is sent from the vehicle's terminal to a server, which then converts the voice data into text using voice recognition software (e.g., Google Speech Recognition).

[2066] 3. Emotion recognition

[2067] The server uses the converted text data and the original voice data to input into an emotion recognition engine, which identifies the user's emotions from the tone of the voice and keywords used. This engine implements natural language processing and machine learning algorithms.

[2068] 4. Response Generation

[2069] The server generates an appropriate response based on the emotional information obtained by the emotion recognition engine. A generative AI model is used to generate the response. This AI model considers the emotional information and the content of the text data to propose the optimal response text.

[2070] 5. Speech Synthesis

[2071] The response text is converted to voice data on the server using Text-to-Speech (TTS) technology (e.g., pyttsx3), which mimics human voice tones and communicates to the user with natural pronunciation.

[2072] 6. Audio playback

[2073] The generated audio data is played back through the vehicle's speakers, allowing the user to hear the system's response.

[2074] Example of a system

[2075] For example, if a user in a self-driving vehicle says, "Turn left at the next intersection," the system will:

[2076] 1. Acquiring voice input

[2077] The user's instruction, "Turn left at the next intersection," is picked up by a microphone inside the car.

[2078] 2. Voice Recognition

[2079] The in-car device sends the voice data to the server, and Google Speech Recognition converts this data into text such as "Turn left at the next intersection."

[2080] 3. Emotion recognition

[2081] The server uses this text data and voice data to have an emotion recognition engine determine that the user is "confused."

[2082] 4. Response Generation

[2083] The server uses a generative AI model to generate a response based on emotion and text: "Okay, turn left at the next intersection."

[2084] 5. Speech Synthesis

[2085] The server uses text-to-speech technology to convert the generated response text into voice data.

[2086] 6. Audio playback

[2087] The response "Okay, turn left at the next intersection" is played over the car's speakers.

[2088] Prompt Sentence Examples

[2089] 001: "Turn left at the next intersection."

[2090] 002: System interprets the user's tone as confused and gives the response: "Okay. Turn left at the next intersection."

[2091] The system can accurately recognize a user's emotions and generate quick and appropriate responses based on those emotions, significantly improving communication for hearing-impaired users.

[2092] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2093] Step 1:

[2094] The user provides voice input, for example, "Turn left at the next intersection." The voice input is captured as digital voice data by a microphone inside the vehicle.

[2095] Step 2:

[2096] The device sends the acquired audio data to the server using an HTTP request or a dedicated communication protocol, and the server receives the data.

[2097] Step 3:

[2098] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech Recognition). For example, it generates text data such as "Turn left at the next intersection."

[2099] Step 4:

[2100] The server receives the text and voice data and inputs it into an emotion recognition engine, which uses natural language processing and machine learning algorithms to identify the user's emotion. For example, it outputs emotional information such as "confused."

[2101] Step 5:

[2102] The server uses a generative AI model based on emotion information and text data to generate an appropriate response. For example, in response to the emotion "confusion," it generates the response text "I understand. Turn left at the next intersection."

[2103] Step 6:

[2104] The server converts the generated response text into voice data using Text-to-Speech (TTS) technology (e.g., pyttsx3). For example, it generates voice data that says, "Okay, turn left at the next intersection."

[2105] Step 7:

[2106] The server sends the generated audio data to the device using an HTTP response, and the device receives the data.

[2107] Step 8:

[2108] The device then plays the received voice data through the car's speakers, allowing the user to hear the voice saying, "Okay, turn left at the next intersection."

[2109] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2110] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2111] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2112] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2113] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2114] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2115] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2116] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2117] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2118] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2119] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2120] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2121] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2122] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2123] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2124] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2125] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2126] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2127] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2128] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2129] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2130] The following is further disclosed regarding the above embodiment.

[2131] (Claim 1)

[2132] a means for obtaining a voice input;

[2133] means for converting the voice input into text data;

[2134] means for analyzing the text data and generating an appropriate response text;

[2135] means for converting the response text into voice data;

[2136] means for reproducing the audio data;

[2137] A system including:

[2138] (Claim 2)

[2139] means for receiving the audio data and converting it into text data for analysis;

[2140] artificial intelligence-based means for analyzing said text data and generating an appropriate response;

[2141] means for converting the generated response into audio data;

[2142] means for transmitting the audio data;

[2143] 10. The system of claim 1, comprising:

[2144] (Claim 3)

[2145] means for receiving said voice input and creating said voice data;

[2146] a communication means for transmitting and receiving the audio data;

[2147] means for generating a response based on the parsed text;

[2148] 10. The system of claim 1, further comprising means for replaying the response.

[2149] That's all.

[2150] "Example 1"

[2151] (Claim 1)

[2152] a means for obtaining a voice input;

[2153] a pre-processing means for converting the voice input into digital voice data to generate high quality voice data;

[2154] a communication means for transmitting the high-quality voice data to an analysis server;

[2155] means for converting the high-quality voice data into text data;

[2156] means for analyzing the text data and generating an appropriate response text;

[2157] means for converting the response text into voice data;

[2158] means for receiving and playing the audio data;

[2159] A system including:

[2160] (Claim 2)

[2161] means for receiving the voice data and converting it into text data for analysis;

[2162] artificial intelligence-based means for analyzing said text data and generating an appropriate response;

[2163] means for converting the generated response into voice data and outputting the voice data in a form understandable to a user;

[2164] A highly efficient communication means for transmitting and receiving said audio data;

[2165] 10. The system of claim 1, comprising:

[2166] (Claim 3)

[2167] means for receiving said voice input and producing high quality digital voice data;

[2168] a communication means for transmitting the voice data to a server and receiving analyzed text data;

[2169] A means using a generative AI model to generate an appropriate response based on the text data;

[2170] means for converting the response into audio data and playing it back;

[2171] 10. The system of claim 1, comprising:

[2172] "Application Example 1"

[2173] (Claim 1)

[2174] A system to assist people with hearing limitations in communicating in their daily lives, comprising:

[2175] a means for obtaining a voice input;

[2176] means for converting the voice input into text data;

[2177] means for analyzing the text data and generating an appropriate response text;

[2178] means for converting the response text into voice data;

[2179] means for reproducing the audio data;

[2180] means including an artificial intelligence model for processing said text data and audio data;

[2181] A means for executing the system on a smart device (smartphone, smart glasses, etc.) installed in a physical store;

[2182] A means for using the voice data to facilitate communication in a physical store;

[2183] A system including:

[2184] (Claim 2)

[2185] means for receiving the audio data and converting it into text data for analysis;

[2186] artificial intelligence-based means for analyzing said text data and generating an appropriate response;

[2187] means for converting the generated response into audio data;

[2188] means for transmitting the audio data;

[2189] A means functioning as an application installed on a smart device in a physical store;

[2190] 10. The system of claim 1, comprising:

[2191] (Claim 3)

[2192] means for receiving said voice input and creating said voice data;

[2193] a communication means for transmitting and receiving the audio data;

[2194] means for generating a response based on the parsed text;

[2195] means for reproducing said response;

[2196] 10. The system of claim 1 intended for use in a brick-and-mortar store.

[2197] "Example 2: Combining Emotion Engines"

[2198] (Claim 1)

[2199] a means for obtaining a voice input;

[2200] means for converting the voice input into text data;

[2201] means for analyzing the text data and voice data and recognizing the user's emotion;

[2202] means for generating an appropriate response text based on the emotion information;

[2203] means for converting the response text into voice data;

[2204] means for reproducing the audio data;

[2205] means for transmitting and receiving the audio data;

[2206] A system including:

[2207] (Claim 2)

[2208] means for receiving the audio data and converting it into text data for analysis;

[2209] artificial intelligence-based means for analyzing said text data and generating an appropriate response;

[2210] means for recognizing a user's emotion by inputting the text data and voice data;

[2211] means for converting the generated response into audio data;

[2212] means for transmitting the audio data;

[2213] 10. The system of claim 1, comprising:

[2214] (Claim 3)

[2215] means for receiving said voice input and creating said voice data;

[2216] a communication means for transmitting and receiving the audio data;

[2217] means for generating a response using the user's emotion information based on the text data and the voice data;

[2218] 10. The system of claim 1, further comprising means for replaying the response.

[2219] "Application example 2 when combining emotion engines"

[2220] (Claim 1)

[2221] a means for obtaining a voice input;

[2222] means for converting the voice input into text data;

[2223] means for recognizing emotions from the text data and voice data;

[2224] means for analyzing the text data in consideration of the emotion and generating an appropriate response text;

[2225] means for converting the response text into voice data;

[2226] means for reproducing the audio data;

[2227] A system including:

[2228] (Claim 2)

[2229] means for receiving the audio data and converting it into text data for analysis;

[2230] an artificial intelligence based means for analyzing the text data and emotion data and generating an appropriate response;

[2231] means for converting the generated response into audio data;

[2232] means for transmitting the audio data;

[2233] 10. The system of claim 1, comprising:

[2234] (Claim 3)

[2235] means for receiving said voice input and creating said voice data;

[2236] a communication means for transmitting and receiving the voice data and emotion data;

[2237] means for generating a response based on the parsed text and sentiment information;

[2238] 10. The system of claim 1, further comprising means for replaying the response. [Explanation of symbols]

[2239] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining a voice input; means for converting the voice input into text data; means for analyzing the text data and generating an appropriate response text; means for converting the response text into voice data; means for reproducing the audio data; A system including:

2. means for receiving the audio data and converting it into text data for analysis; artificial intelligence-based means for analyzing said text data and generating an appropriate response; means for converting the generated response into audio data; means for transmitting the audio data; The system of claim 1 , comprising:

3. means for receiving said voice input and creating said voice data; a communication means for transmitting and receiving the audio data; means for generating a response based on the parsed text; 2. The system of claim 1, further comprising means for replaying said response. That's all.

Images