System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving voice data; means for converting the received voice data into text data; means for transmitting the converted text data; means for receiving text data input by a user; means for converting the received text data into voice data; and means for transmitting the converted voice data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a problem in that it is difficult to seamlessly convert speech and text between hearing-impaired and hearing-normal individuals and communicate in real time. Conventional technologies lack effective means for facilitating communication between the two, making it difficult for dialogue to occur naturally. For example, hearing-impaired individuals require subtitle services to understand speech, but providing such services in real time is difficult. Furthermore, the process of converting text to speech so that hearing-normal individuals can understand the text of the hearing-impaired is also complicated. To solve these problems, the present invention aims to provide an efficient system for natural dialogue between hearing-impaired and hearing-normal individuals. [Means for solving the problem]

[0005] The present invention provides a system including a means for receiving voice data, a means for converting the received voice data into text data, a means for transmitting the converted text data, a means for receiving text data entered by a user, a means for converting the received text data into voice data, and a means for transmitting the converted voice data. This system can convert voice input provided by a hearing-impaired person into text in real time and display it to a hearing-impaired person, and can convert text input by the hearing-impaired person into voice in real time and play it back to the hearing-impaired person. This enables seamless communication between hearing-impaired people and hearing-able people, allowing for natural dialogue.

[0006] "Voice Data" means a digital representation of a user's speech.

[0007] "Means for receiving" refers to a device or method for obtaining data from the outside.

[0008] "Means for converting" refers to a device or method for converting one data format to another data format.

[0009] "Text data" refers to data that represents character information in digital form.

[0010] "Transmitting means" refers to a device or method for transferring data to another device or system.

[0011] "User" means any individual who uses the System.

[0012] "Means for input" refers to a device or method by which a user provides data to the system.

[0013] "System" refers to an entire device or mechanism in which multiple elements work together. [Brief explanation of the drawings]

[0014] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of hearing-impaired people into text data, and converts the text data input by hearing-impaired people into voice data, enabling natural dialogue in real time. Here, the detailed operation of this system is explained in natural language.

[0036] Overall system configuration

[0037] The system consists of the following main components:

[0038] Server: The generation AI and chat generation AI run, converting voice data and text data.

[0039] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0040] Users: Deaf and hearing impaired people who use the system.

[0041] Processing voice input

[0042] First, a person with normal hearing speaks into the device to input voice. When a person with normal hearing speaks, "Hello. How are you today?", the voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0043] Speech-to-text conversion

[0044] The voice data received by the server is processed by the ASR (Automatic Speech Recognition) engine, a component of the generative AI. This engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?" This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[0045] Text entry for the hearing impaired

[0046] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling great." This input text data is sent to the server by the device.

[0047] Text to speech conversion

[0048] The text data received by the server is processed by a TTS (Text to Speech) engine, a component of the chat generation AI. This engine analyzes the text data and generates the corresponding voice data, "Hello. The weather is nice today and I'm in a good mood." This generated voice data is sent to the device of the able-bodied person and played back through the speaker.

[0049] Specific examples

[0050] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0051] 1. Voice input from a healthy subject:

[0052] A healthy person speaks into the device: "Hello. How are you today?"

[0053] The terminal transmits the voice data to the server.

[0054] The server converts the voice data into text "Hello, how are you today?"

[0055] The text appears on the device of the hearing impaired person.

[0056] 2. Text Response for the Deaf:

[0057] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[0058] The terminal transmits the text data to the server.

[0059] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[0060] The device of the able-bodied person plays the audio.

[0061] This system will enable natural and seamless communication between hearing-impaired and hearing-suffering people. This invention will significantly improve the communication environment for hearing-impaired people, enabling both parties to communicate efficiently.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] A user (with normal hearing) speaks to the terminal. The voice input device captures the user's speech, "Hello. How are you today?" and processes it as digital voice data.

[0065] Step 2:

[0066] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[0067] Step 3:

[0068] The server decodes the voice data received from the device and passes it to the ASR (Automatic Speech Recognition) engine for processing. The ASR engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?"

[0069] Step 4:

[0070] The server encodes the generated text data and transmits it over the network to the hearing-impaired user's device. The text data is packetized and transmitted according to the appropriate communication protocol.

[0071] Step 5:

[0072] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The hearing impaired person reads and understands this text.

[0073] Step 6:

[0074] A user (who is hearing impaired) uses the text input device of the device to input a reply, for example, "Hello. The weather is nice today and I'm in a good mood."

[0075] Step 7:

[0076] The terminal receives the input text data and transmits it to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[0077] Step 8:

[0078] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine for processing. The TTS engine analyzes the text data and generates the corresponding voice data: "Hello. The weather is nice today and I'm in a good mood."

[0079] Step 9:

[0080] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[0081] Step 10:

[0082] The device (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. The weather is nice today and I'm feeling great." using the device's speaker. A able-bodied person listens to and understands this voice.

[0083] These are the specific processing steps of the system, which allows for seamless communication between hearing-impaired and hearing-savvy people.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] In communication between hearing-impaired and hearing-savvy people, there is a need for an efficient and accurate system that can convert speech to text and vice versa in real time to achieve seamless dialogue. Conventional systems have insufficient conversion accuracy and speed, often causing stress and discomfort during actual dialogue. In addition, the need to use multiple different devices and applications makes operation cumbersome, and there have been few easy-to-use systems.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, means for transmitting the converted text data, means for receiving text data entered by a user, means for converting the received text data into voice data, means for transmitting the converted voice data, means including an automatic speech recognition engine using a generative AI model, and means including a text-to-speech synthesis engine using a generative AI model, thereby enabling real-time and seamless dialogue between hearing-impaired people and hearing-normal people.

[0089] "Audio data" is data that represents an audio signal in digital form.

[0090] "Text data" is data that represents character information in digital form.

[0091] A "generative AI model" is an artificial intelligence model used to automatically generate and transform data.

[0092] An "automatic speech recognition engine" is a software component that analyzes speech data and converts it into corresponding text data.

[0093] A "text-to-speech engine" is a software component that analyzes text data and converts it into corresponding speech data.

[0094] "Network connectivity" is the infrastructure for communicating data between devices.

[0095] "User" refers to an individual who uses the system, and includes both hearing-impaired and hearing-nondisabled individuals.

[0096] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of the hearing-impaired person into text data, and converts the text data input by the hearing-impaired person into voice data, enabling natural dialogue in real time.

[0097] Overall system configuration

[0098] The system consists of the following main components:

[0099] Server: The generative AI model and its components, the automatic speech recognition engine (ASR) and the text-to-speech synthesis engine (TTS), run here, converting voice data and text data.

[0100] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0101] Users: Deaf and hearing impaired people who use the system.

[0102] Hardware and software used

[0103] Hardware: Various devices such as a microphone, speaker, display, keyboard, and network interface are required. Examples include smartphones, tablets, and PCs.

[0104] Software: Includes software components such as generative AI models, ASR engines (e.g., Google® Cloud Speech-to-Text, Microsoft® Azure® Speech Services), and TTS engines (e.g., Google Cloud Text-to-Speech, AWS® Polly).

[0105] Data processing and data calculation

[0106] The server receives the voice data sent from the device, analyzes it with an ASR engine, and converts it into text data. This text data is then sent back to the device and displayed on the screen. Conversely, the text data sent from the device is converted into voice data by the server with a TTS engine, and played back on the device. This series of processes enables seamless communication between hearing-impaired and hearing-normal people.

[0107] Specific examples

[0108] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0109] 1. Voice input from a healthy subject:

[0110] A healthy person speaks into the device: "Hello. How are you today?"

[0111] The terminal transmits the voice data to the server.

[0112] The server converts the voice data into text "Hello, how are you today?"

[0113] The text appears on the device of the hearing impaired person.

[0114] 2. Text Response for the Deaf:

[0115] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[0116] The terminal transmits the text data to the server.

[0117] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[0118] The device of the able-bodied person plays the audio.

[0119] Prompt Sentence Examples

[0120] "A hearing-impaired person inputs speech, which is converted into text and displayed to a hearing-impaired person. Please explain in detail the system that inputs this text, converts it back into speech, and plays it back to the hearing-impaired person."

[0121] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people. By utilizing generative AI models, highly accurate and real-time dialogue is possible, significantly removing communication barriers.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] (Details of processing steps)

[0124] Step 1:

[0125] Voice input and capture for able-bodied people

[0126] Input: A healthy person speaks into the terminal.

[0127] Specific action: A healthy person says, "Hello. How are you today?"

[0128] Data processing and output: The microphone on the device captures the audio and stores it in a temporary buffer as digital audio data. This audio data is then sent to the server via the network.

[0129] Step 2:

[0130] Speech-to-text conversion

[0131] Input: Digital audio data sent from the device.

[0132] Specific operation: The server receives the audio data.

[0133] Data processing and output: An automatic speech recognition engine (ASR), part of the generative AI model on the server, analyzes the voice data and converts it into corresponding text data: "Hello. How are you today?" This text data is stored in a temporary database and sent to the hearing-impaired person's device.

[0134] Step 3:

[0135] Displaying Text

[0136] Input: Text data sent from the server.

[0137] Specific operation: The hearing-impaired person's device receives text data.

[0138] Data processing and output: Displaying text on the terminal display. For example, updating the text "Hello, how are you today?" in the display area.

[0139] Step 4:

[0140] Text entry for the hearing impaired

[0141] Input: A hearing-impaired person types text into a terminal.

[0142] Specific action: A hearing-impaired person types, "Hello. The weather is nice today and I'm feeling great."

[0143] Data processing and output: The input text data is temporarily stored on the terminal and sent to the server via the network.

[0144] Step 5:

[0145] Text to speech conversion

[0146] Input: Text data sent from the terminal.

[0147] Specific operation: The server receives the text data.

[0148] Data processing and output: A text-to-speech engine (TTS), part of the generative AI model on the server, analyzes the text data and converts it into corresponding voice data: "Hello. The weather is nice today and I'm feeling great." This voice data is stored in a temporary database and sent to the able-bodied user's device.

[0149] Step 6:

[0150] Playing audio

[0151] Input: Audio data sent from the server.

[0152] Specific operation: The device of the healthy person receives the voice data.

[0153] Data processing and output: The received voice data is played back through the device's speaker. For example, a healthy person hears a voice saying, "Hello. The weather is nice today and I'm feeling good."

[0154] By inputting, processing, and outputting data at each step, seamless real-time communication between hearing-impaired and hearing-speaking people is realized. The system uses speech recognition and speech synthesis technology that utilizes generative AI models to provide high accuracy and smooth dialogue.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] In modern society, there are limited ways for hearing-impaired and hearing-normal people to communicate effectively. In particular, in brick-and-mortar stores, hearing-impaired customers often have difficulty communicating with store staff, which can affect the quality of service. To solve this problem, a system is needed that allows both hearing-impaired and hearing-normal people to communicate naturally and efficiently.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes a means for receiving voice data, a means for converting the received voice data into text data, and a means for transmitting the converted text data, thereby enabling two-way communication of text data and voice data between customers and store clerks in a physical store.

[0160] The "means for receiving audio data" is a device or function that acquires audio input from the outside as digital data.

[0161] The "means for converting received voice data into text data" refers to a device or function that analyzes voice data and converts it into corresponding text data.

[0162] The "means for transmitting converted text data" refers to a device or function that transmits the generated text data to another device or system.

[0163] The "means for receiving text data entered by a user" refers to a device or function that acquires text data entered by a user.

[0164] The "means for converting received text data into voice data" refers to a device or function that analyzes text data and converts it into corresponding voice data.

[0165] The "means for transmitting converted audio data" is a device or function that transmits the generated audio data to another device or system.

[0166] "Means for enabling two-way communication of text data and voice data between customers and store clerks in a physical store" refers to a device or function that allows customers and store clerks in a physical store to exchange information in text and voice format two-way.

[0167] This invention is a system that enables seamless communication between hearing-impaired people and hearing-normal people (store clerks) in brick-and-mortar stores. This system converts voice data into text data and text data into voice data, enabling natural dialogue in real time.

[0168] Overall system configuration

[0169] The system consists of the following main components:

[0170] Server: The server runs a speech recognition engine, a text-to-speech synthesis engine, and a data transmission / reception module, and converts voice data into text data.

[0171] Terminal: A device used by users (hearing impaired and hearing-able) that is capable of voice input / output, and text input / display, such as a smartphone.

[0172] Users: Deaf and hearing-impaired people (store clerks) who use the system.

[0173] Processing voice input

[0174] First, a person with normal hearing speaks into the device to input voice. For example, if a store clerk says, "Are you looking for something?", this voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0175] Speech-to-text conversion

[0176] The voice data received by the server is analyzed by a voice recognition engine, and the corresponding text data, "Are you looking for something?", is generated. This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[0177] Text entry for the hearing impaired

[0178] Next, the hearing-impaired person uses their own device to input a reply, for example, "Yes, I'm looking for shoes." This input text data is sent to the server by the device.

[0179] Text to speech conversion

[0180] The text data received by the server is analyzed by a text-to-speech synthesis engine, which generates the corresponding voice data, "Hello, I'm looking for shoes." This voice data is sent to the device of the able-bodied person and played back through a speaker.

[0181] Specific examples

[0182] For example, here is a concrete example of a conversation between a hearing person and a hearing-impaired person:

[0183] 1. Voice input from a healthy subject:

[0184] Store clerk: "Are you looking for something?"

[0185] The terminal transmits the voice data to the server.

[0186] The server converts the voice data into the text "Are you looking for something?"

[0187] The text appears on the device of the hearing impaired person.

[0188] 2. Text Response for the Deaf:

[0189] Deaf person: "Yes, I'm looking for my shoes."

[0190] The terminal transmits the text data to the server.

[0191] The server converts the text data into speech, "Yes, I'm looking for shoes."

[0192] The clerk's terminal plays the audio.

[0193] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people in brick-and-mortar stores. This invention significantly improves the environment in which hearing-impaired people receive services in brick-and-mortar stores, enabling both parties to communicate efficiently.

[0194] Prompt Sentence Examples

[0195] "Hello. How are you today? Is there anything I can help you with?"

[0196] The above configuration provides an environment in which hearing-impaired and hearing-normal people can communicate smoothly in physical stores.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] Voice input is performed when a person with hearing loss speaks into the device. The device captures this voice input as digital voice data. The input is what the person with hearing loss said (voice), and the output is digital voice data. This digital voice data is temporarily stored in the device.

[0200] Step 2:

[0201] The terminal transmits the captured audio data to the server via the network. The input is the captured digital audio data, and the output is the audio data transmitted to the server. This is where the data transmission module operates.

[0202] Step 3:

[0203] The speech recognition engine analyzes the voice data received by the server and converts it into corresponding text data. The input is the received digital voice data and the output is the corresponding text data. The speech recognition engine uses a generative AI model to analyze the voice and generate accurate text.

[0204] Step 4:

[0205] The server sends the generated text data to the terminal of the hearing impaired person. The input is the converted text data, and the output is the text data sent to the terminal of the hearing impaired person. Here, the data sending module is operated again.

[0206] Step 5:

[0207] The hearing impaired person's device displays the received text data on the screen. The input is the received text data, and the output is the text displayed on the display screen. The display module of the device operates to provide the text visually.

[0208] Step 6:

[0209] A hearing-impaired person types a text reply into their device. The input is text data entered by the hearing-impaired person, and the output is text data stored on the device. This text is entered using the device's input device (keyboard or touch screen).

[0210] Step 7:

[0211] The terminal sends the input text data to the server. The input is the text data entered by the hearing impaired person, and the output is the text data sent to the server. The terminal transfers the data via the network.

[0212] Step 8:

[0213] The text data received by the server is analyzed by the text-to-speech engine and converted into corresponding voice data. The input is the received text data and the output is the generated voice data. The text-to-speech engine uses a generative AI model to generate natural-sounding voices.

[0214] Step 9:

[0215] The server sends the generated voice data to the device of the able-bodied person. The input is the generated voice data, and the output is the voice data sent to the device of the able-bodied person. Here too, the data transmission module operates.

[0216] Step 10:

[0217] The device of the able-bodied person plays the received voice data. The input is the received voice data, and the output is the voice to be played. The speaker module of the device operates and provides the voice to the able-bodied person.

[0218] These are the specific processing steps in this system. At each step, data is processed and calculated, ultimately achieving natural conversation between hearing-impaired and hearing-normal people.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts the voice input of hearing-impaired people into text data and the text data input of hearing-impaired people into voice data, but also analyzes and conveys emotional information, enabling more natural and rich communication. Here, the detailed operation of this system is explained in natural language.

[0221] Overall system configuration

[0222] The system consists of the following main components:

[0223] Server: The generation AI, chat generation AI, and emotion engine run, converting voice data and text data.

[0224] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0225] Users: Deaf and hearing impaired people who use the system.

[0226] Voice input and emotion recognition

[0227] First, a person with normal hearing speaks into the device to input voice. "Hello. How are you today?" the person says. This voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0228] The voice data received by the server is analyzed by an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. The emotional information recognized by the emotion engine is added as a tag to the converted text data. For example, it may be in the form of "Hello. How are you today? (Joy)."

[0229] Displaying text data and conveying emotions

[0230] The emotion-tagged text data generated by the server is sent over the network to the hearing-impaired person's device. The device decodes the received text data and displays "Hello. How are you today?" on the screen, along with visually displaying emotion information using icons and colors that indicate the emotion tag. This allows the hearing-impaired person to understand the emotions of people without hearing.

[0231] Text input and emotion communication for the hearing impaired

[0232] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This input text data is sent to the server along with the emotion information.

[0233] Text-to-speech and emotional response

[0234] The text data received by the server is passed to a TTS (Text to Speech) engine along with emotional information. The TTS engine analyzes the text data and generates voice data, "Hello. It's nice weather today and I'm feeling great," based on the emotional information. This voice data is given an intonation that corresponds to the emotional information. For example, if the emotion tag is "joy," a lively and cheerful voice is generated.

[0235] Specific examples

[0236] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0237] 1. Speech input and emotion recognition for healthy subjects:

[0238] A healthy person speaks into the device: "Hello. How are you today?" (joy)

[0239] The terminal transmits the voice data to the server.

[0240] The server analyzes the voice data and recognizes the emotion as "joy."

[0241] The voice data is converted into text "Hello. How are you today?" and an emotion tag (joy) is assigned.

[0242] Text and emotional information is sent to the device of a hearing-impaired person and displayed visually.

[0243] 2. Text replies and emotional reflection for the deaf:

[0244] A deaf person types text into a device: "Hello. The weather is nice today and I'm feeling great." (Joy)

[0245] The device sends text data and emotion information to the server.

[0246] The server analyzes the text data and generates a voice message based on the emotional information: "Hello. The weather is nice today and I'm in a good mood."

[0247] The emotion of "joy" is reflected in the audio data and played back on a device belonging to a healthy person.

[0248] This system will enable more natural and rich communication between hearing-impaired and hearing-suffering people, incorporating emotional information. The introduction of an emotion engine will enable dialogue to go beyond simply exchanging information, and involve the transmission of emotions, deepening mutual understanding.

[0249] The processing flow will be explained below.

[0250] Step 1:

[0251] A user (with normal hearing) speaks to the device. The voice input device captures the user's speech and processes it as digital voice data.

[0252] Step 2:

[0253] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[0254] Step 3:

[0255] The server decodes the voice data received from the device and passes it to the emotion engine, which analyzes the voice data and recognizes the user's emotion (e.g., "happiness," "sadness," "anger," etc.).

[0256] Step 4:

[0257] The server receives the emotion information from the emotion engine and passes the voice data to the ASR (Automatic Speech Recognition) engine, which analyzes the voice data and generates corresponding text data (e.g., "Hello. How are you today?").

[0258] Step 5:

[0259] The server tags the generated text data with emotional information recognized by the emotion engine (e.g., "Hello. How are you today? (Joy)").

[0260] Step 6:

[0261] The server encodes the text data with emotion tags and sends it over the network to the hearing-impaired person's device. The text data is packetized and sent according to the appropriate communication protocol.

[0262] Step 7:

[0263] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The emotion tag (joy) is visually displayed using icons and colors.

[0264] Step 8:

[0265] A user (who is hearing impaired) uses the device's text input device to type a reply (e.g., "Hello. I'm in a good mood today.") and also types an emotion tag (joy).

[0266] Step 9:

[0267] The device receives the input text data and emotion tag, and transmits them to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[0268] Step 10:

[0269] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine based on the emotion tag. The TTS engine analyzes the text data and generates corresponding voice data (e.g., "Hello. I'm in a good mood today because the weather is nice.").

[0270] Step 11:

[0271] The server adds emotional intonation to the voice data based on the emotion tag (joy), generating voice data that reflects the emotion.

[0272] Step 12:

[0273] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[0274] Step 13:

[0275] The terminal (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. It's nice weather today and I'm feeling good" using a speaker. The voice reflects emotional intonation, and able-bodied people can understand this voice.

[0276] These are the specific processing steps of a system that combines an emotion engine. This processing enables more natural and richer dialogue that includes emotional information.

[0277] Example 2

[0278] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0279] Conventional communication systems have difficulty in achieving natural dialogue, including emotional content, between hearing-impaired and hearing-challenged people. Simply converting the speech input of hearing-impaired people into text and the text input of hearing-impaired people into speech results in the loss of emotional nuances and the resulting dialogue becoming monotonous. This drawback is particularly serious in communication where sharing emotions is important, making it difficult for people to understand each other's emotions.

[0280] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing emotional information from the received voice data, means for converting the voice data with added emotional information into text data, means for transmitting the converted text data and emotional information, means for receiving text data entered by the user together with the emotional information, means for converting the received text data and emotional information into voice data, and means for transmitting the converted voice data and emotional information. This enables more natural and rich communication, including emotion, between hearing-impaired people and those with hearing disabilities.

[0281] The "means for receiving voice data" refers to a mechanism for inputting voice data generated from an input device of a healthy person into a server.

[0282] "Means for analyzing emotion information" refers to a mechanism for identifying the user's emotion from received voice data using an emotion engine and extracting that information.

[0283] "Means for converting voice data into text data" refers to a mechanism for converting received and analyzed voice data into text information via an ASR (automatic speech recognition) engine.

[0284] "Means for transmitting text data to which emotional information has been added" refers to a mechanism for adding emotional information to converted text data and transmitting it to another terminal via a network.

[0285] The "means for receiving text data entered by a user together with emotion information" refers to a mechanism for capturing the character information entered by a user into a terminal and the selected emotion tag into a server.

[0286] "Means for converting text data and emotional information into voice data" refers to a mechanism for converting received text data and emotional information into voice data via a TTS (Text to Speech Synthesis) engine.

[0287] The "means for transmitting the converted voice data and emotional information" refers to a mechanism for transmitting the generated voice data and its emotional information to another terminal via a network.

[0288] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts voice data into text data but also analyzes and transmits emotional information, enabling more natural and rich communication.

[0289] Voice input and emotion recognition

[0290] When a user (a person with normal hearing) speaks into the device, their voice is captured using a microphone and processed as digital voice data. For example, a person with normal hearing might say, "Hello. How are you today?" This voice data is sent to a server via a network.

[0291] Audio data processing on the server

[0292] The server analyzes the received voice data using an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. Emotional information is added to the converted text data as a tag. For example, it might look like "Hello. How are you today? (Joy)". This generates data that includes text data and emotional information.

[0293] Displaying text data and conveying emotions

[0294] The device receives the emotion-tagged text data sent from the server and displays it on the screen, while visually displaying emotion information using icons and colors that indicate the emotion tag. This allows hearing-impaired people to understand the emotions of hearing-impaired people.

[0295] Text input and emotion communication for the hearing impaired

[0296] Next, the hearing-impaired person uses their own device to input a reply text. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This text data and emotional information are sent to the server via the network.

[0297] Text data processing on the server

[0298] The text data received by the server is passed to a TTS (Text to Speech) engine. The TTS engine analyzes the text data and generates voice data based on the emotional information. For example, if an emotional tag (joy) is attached, a lively and cheerful voice saying "Hello. The weather is nice today and I'm in a good mood" is generated.

[0299] Playing audio data

[0300] The generated voice data is sent to the device of the hearing-impaired person and played back, allowing the hearing-impaired person to understand the emotions of the hearing-impaired person.

[0301] Specific examples

[0302] Here are some examples of specific prompts:

[0303] 1. "Turn the following speech data into text and analyze the user's sentiment: Hello. How are you today?"

[0304] 2. "Generate speech from emotion-tagged text data: Hello. The weather is nice today and I'm feeling great. (Joy)"

[0305] Hardware and software used

[0306] Devices: Smartphones, tablets, PCs, etc.

[0307] Server: Generation AI, chat generation AI, emotion engine, ASR engine, TTS engine

[0308] Emotion Recognition Software: Emotion Engine

[0309] Automatic Speech Recognition Software: ASR Engine

[0310] Text to Speech Software: TTS Engine

[0311] By using this system, communication between hearing-impaired and hearing-challenged people can take place in a way that includes emotional information, enabling a more natural and deeper mutual understanding.

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] Device:

[0315] A user (healthy person) speaks into the device. Example: "Hello. How are you today?"

[0316] Input: Normal human speech

[0317] How it works: The device's microphone captures audio and converts it into audio data.

[0318] Output: Digital audio data

[0319] Step 2:

[0320] Device:

[0321] Input: Captured digital audio data

[0322] What it does: The device sends this digital audio data to the server.

[0323] Output: Audio data sent to the server

[0324] Step 3:

[0325] server:

[0326] Input: Transmitted audio data

[0327] How it works: The server analyzes this voice data through an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger).

[0328] Output: Audio data with emotional information added

[0329] Step 4:

[0330] server:

[0331] Input: Audio data with emotional information

[0332] How it works: This data is converted into text using an ASR (automatic speech recognition) engine.

[0333] Output: Emotion-tagged text data (e.g., "Hello. How are you today? (Joy)")

[0334] Step 5:

[0335] server:

[0336] Input: Emotion-tagged text data

[0337] How it works: The server sends the generated emotion-tagged text data to the hearing-impaired person's device via the network.

[0338] Output: Text data and emotional information sent to the hearing-impaired person's device

[0339] Step 6:

[0340] Terminal (hearing impaired):

[0341] Input: Emotion-tagged text data

[0342] How it works: The device displays the received text data on the screen and visually represents the emotional information using icons and colors.

[0343] Output: Text and emotion information displayed on the screen

[0344] Step 7:

[0345] User (hearing impaired):

[0346] Input: Terminal input field

[0347] Action: A deaf person types a reply. Example: "Hello. The weather is nice today and I'm feeling great."

[0348] Output: Input text data

[0349] Step 8:

[0350] Terminal (hearing impaired):

[0351] Input: Text data entered by a hearing-impaired person

[0352] What it does: Sends the text and the selected emotion tag to the server.

[0353] Output: Text data and emotion information sent to the server

[0354] Step 9:

[0355] server:

[0356] Input: Received text data and emotion information

[0357] How it works: The server passes this to a TTS (Text to Speech) engine, which converts the received text data into voice data. At that time, it adds an appropriate tone to the voice based on the emotion tag. For example, for the (joy) tag, a lively and cheerful tone

[0358] Output: Audio data reflecting emotional information

[0359] Step 10:

[0360] server:

[0361] Input: Voice data reflecting emotional information

[0362] Operation: The server transmits the generated voice data to the device of the healthy person via the network.

[0363] Output: Audio data sent to the healthy subject's device

[0364] Step 11:

[0365] Device (healthy person):

[0366] Input: Transmitted audio data

[0367] Action: Plays back the audio data and has a normal subject listen to it. Example: "Hello. The weather is nice today and I'm feeling great."

[0368] Output: Played audio data

[0369] Through these steps, natural communication, including emotional content, is realized between hearing-impaired and hearing-challenged people.

[0370] (Application example 2)

[0371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] It is difficult to achieve smooth and natural communication between hearing-impaired and hearing-savvy people. Conventional systems only convert voice data to text data, and lack emotional exchange in communication. This leads to insufficient communication and a decline in the quality of communication. Especially in brick-and-mortar stores, there is a strong demand for communication that includes emotional information, as this has a significant impact on customer experience and service quality.

[0373] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, emotion analysis means for analyzing the user's emotion and generating emotion information, means for assigning the generated emotion information to the text data, means for visualizing the assigned emotion information, and means for reflecting the assigned emotion information in the voice data. This enables natural and rich communication including emotion information between hearing-impaired and hearing-normal people.

[0374] "Means for receiving audio data" refers to a device or method that receives an external audio signal with low loss and converts it into a format suitable for internal processing.

[0375] A "means for converting received voice data into text data" is a device or method that uses voice recognition technology to convert received voice signals into a corresponding text format.

[0376] The "means for transmitting converted text data" refers to a device or method for transmitting the converted text data to another device or system.

[0377] The "means for receiving text data entered by a user" refers to a device or method that receives text data entered by a user and converts it into a format suitable for internal processing.

[0378] A "means for converting received text data into speech data" is a device or method that uses text-to-speech technology to convert text data into a corresponding speech format.

[0379] The "means for transmitting converted audio data" refers to a device or method for transmitting the converted audio data to another device or system.

[0380] "Emotion analysis means" refers to a device or method that reads the user's emotions from input voice or text data and generates emotion information.

[0381] The "means for adding generated emotion information to text data" refers to a device or method for adding emotion information obtained by emotion analysis to text and reinforcing the text data.

[0382] The "means for visualizing assigned emotional information" is a device or method for visualizing analyzed emotional information and presenting it to the user.

[0383] The "means for reflecting the added emotional information in the voice data" refers to a device or method for incorporating the generated emotional information into the voice synthesis process and expressing it as voice.

[0384] Overall system configuration

[0385] This system aims to realize seamless communication between hearing-impaired and hearing-suffering people. The system is broadly composed of the following elements:

[0386] Server: The emotion analysis, speech recognition, and text-to-speech synthesis engines run on a high-performance cloud server.

[0387] Terminal: A device capable of voice input / output and text display, such as a smartphone or smart glasses.

[0388] Users: Deaf and hearing impaired people who use the system.

[0389] Hardware / Software used

[0390] Server: A cloud server such as AWS EC2 or Google Cloud.

[0391] Emotion engine: Microsoft Azure Emotion API.

[0392] Speech recognition engine (ASR): Google Cloud Speech-to-Text.

[0393] Text-to-speech engine (TTS): Amazon Polly.

[0394] Program processing explanation

[0395] Voice input and emotion recognition

[0396] 1. Terminal: Voice input is performed by a healthy person speaking into the terminal.

[0397] 2. Server: The audio data captured on the device is sent to the server.

[0398] 3. Server (emotion analysis means): The received voice data is analyzed by the emotion engine, and emotional information such as joy, sadness, and anger is generated.

[0399] 4. Server (voice recognition means): Voice data is converted into text data by the ASR engine.

[0400] 5. Server (means for adding emotional information): Emotional information is added to text data as tags. For example, "Hello. How are you today? (Joy)".

[0401] Displaying text data and conveying emotions

[0402] 6. Server: The generated emotion-tagged text data is sent to the hearing-impaired person's device.

[0403] 7. Terminal (visualization means): The text data is decoded and displayed on the screen, and emotion tags are visually displayed using icons and colors.

[0404] Text input and emotion communication for the hearing impaired

[0405] 8. Device: A hearing impaired person uses a device to input text, for example, "Hello. The weather is nice today and I feel good. (Joy)."

[0406] 9. Server: The input text data is sent to the server along with the emotion information.

[0407] Text-to-speech and emotional response

[0408] 10. Server (text-to-speech synthesis means): The received text data is passed to the TTS engine, which analyzes the text data and generates voice data.

[0409] 11. Server (Means for Reflecting Emotional Information): The server adds intonation to the voice data based on the emotional information. For example, if the voice data is tagged with "joy," it generates a voice with a lively and cheerful tone.

[0410] 12. Terminal: The generated voice data is played on the terminal of the able-bodied person.

[0411] Examples of concrete examples and prompts

[0412] Here is a concrete example of the dialogue:

[0413] 1. Healthy person: "Hello. How are you today?" (joy).

[0414] 2. Hearing impaired person: "Hello. The weather is nice today and I feel great." (joy).

[0415] Example prompt for a generative AI model:

[0416] "The store clerk greeted: Hello, is there anything you're looking for?"

[0417] "Customer enters through smart glasses: I'm not looking for anything in particular today."

[0418] "Convert these messages into text and audio with appropriate emotional information."

[0419] This system enables natural and rich communication, including emotional information, between hearing-impaired and hearing-savvy people. Specifically, it is expected to improve the customer experience and service quality in brick-and-mortar stores.

[0420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0421] Step 1:

[0422] Device: Voice input is performed by a non-disabled person speaking into the device. The device's built-in microphone captures the voice signal and stores it as digitally formatted voice data. This voice data is sent to a server for further processing. The input here is the voice spoken by a non-disabled person, and the output is digitally formatted voice data.

[0423] Step 2:

[0424] Server: Receives the voice data sent from the device. The server checks the format of the voice data and converts it to the appropriate format if necessary. At this stage, data pre-processing is performed, such as noise filtering. The input is voice data, and the output is pre-processed voice data.

[0425] Step 3:

[0426] Server (emotion analysis means): Analyzes the received voice data using an emotion engine to generate emotional information for healthy individuals. In the analysis step, emotions are estimated based on the pitch, tone, and speed of the voice. The input is processed voice data, and the output is metadata containing emotional information.

[0427] Step 4:

[0428] Server (voice recognition means): After emotion analysis, the voice data is converted into text data by the ASR engine. The voice recognition engine analyzes the voice waveform and converts it into the corresponding text format. The input here is the voice data that has been emotion analyzed, and the output is the converted text data.

[0429] Step 5:

[0430] Server (emotion information adding means): Adds the generated emotion information to the converted text data as a tag. Specifically, emotion information is added to the end of a sentence in the text. For example, "Hello. How are you today? (Joy)". The input is text data and emotion metadata, and the output is emotion-tagged text data.

[0431] Step 6:

[0432] Server: Sends emotion-tagged text data to the device of the hearing-impaired person. Data transfer takes place here, and stable communication is ensured using network protocols (e.g., HTTP, WebSocket). The input is emotion-tagged text data, and the output is the text data received by the device of the hearing-impaired person.

[0433] Step 7:

[0434] Terminal (Visualization Means): The terminal of the hearing-impaired person decodes the received text data and displays it on the screen. At the same time, visual feedback (e.g., icons or colors) based on the emotion tag is also displayed. The input is emotion-tagged text data, and the output is the displayed text and emotion information.

[0435] Step 8:

[0436] User: A hearing-impaired person uses the device to input a text message. For example, "Hello. The weather is nice today and I feel great. (Joy)." The device receives the user's input and stores it as text data. The input is the text entered by the user, and the output is text data.

[0437] Step 9:

[0438] Terminal: Sends input text data to the server. The terminal's communication module sends the text data to the server, and the data is encrypted if necessary. The input is text data, and the output is the text data sent to the server.

[0439] Step 10:

[0440] Server (Text-to-Speech Synthesis Means): The received text data is converted into voice data by the TTS engine. The TTS engine analyzes the text and generates natural-sounding voice. The input is text data, and the output is the generated voice data.

[0441] Step 11:

[0442] Server (emotional information reflection means): Reflects emotional information in the generated voice data. Specifically, it adjusts the tone and pitch of the voice based on the emotion tag. The input is voice data and emotional information, and the output is voice data that reflects the emotional information.

[0443] Step 12:

[0444] Terminal: The audio data reflecting emotional information is played on the terminal of the hearing-impaired person. Here, the hearing-impaired person can receive a message from the hearing-impaired person by voice. The input is the audio data reflecting emotional information, and the output is the audio played from the terminal.

[0445] In this way, the system can realize natural communication including emotional information between hearing-impaired and hearing-normal people.

[0446] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0447] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0448] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0449] [Second embodiment]

[0450] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0451] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0452] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0453] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0454] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0455] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0456] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0457] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0458] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0459] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0460] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0461] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0462] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of hearing-impaired people into text data, and converts the text data input by hearing-impaired people into voice data, enabling natural dialogue in real time. Here, the detailed operation of this system is explained in natural language.

[0463] Overall system configuration

[0464] The system consists of the following main components:

[0465] Server: The generation AI and chat generation AI run, converting voice data and text data.

[0466] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0467] Users: Deaf and hearing impaired people who use the system.

[0468] Processing voice input

[0469] First, a person with normal hearing speaks into the device to input voice. When a person with normal hearing speaks, "Hello. How are you today?", the voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0470] Speech-to-text conversion

[0471] The voice data received by the server is processed by the ASR (Automatic Speech Recognition) engine, a component of the generative AI. This engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?" This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[0472] Text entry for the hearing impaired

[0473] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling great." This input text data is sent to the server by the device.

[0474] Text to speech conversion

[0475] The text data received by the server is processed by a TTS (Text to Speech) engine, a component of the chat generation AI. This engine analyzes the text data and generates the corresponding voice data, "Hello. The weather is nice today and I'm in a good mood." This generated voice data is sent to the device of the able-bodied person and played back through the speaker.

[0476] Specific examples

[0477] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0478] 1. Voice input from a healthy subject:

[0479] A healthy person speaks into the device: "Hello. How are you today?"

[0480] The terminal transmits the voice data to the server.

[0481] The server converts the voice data into text "Hello, how are you today?"

[0482] The text appears on the device of the hearing impaired person.

[0483] 2. Text Response for the Deaf:

[0484] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[0485] The terminal transmits the text data to the server.

[0486] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[0487] The device of the able-bodied person plays the audio.

[0488] This system will enable natural and seamless communication between hearing-impaired and hearing-suffering people. This invention will significantly improve the communication environment for hearing-impaired people, enabling both parties to communicate efficiently.

[0489] The processing flow will be explained below.

[0490] Step 1:

[0491] A user (with normal hearing) speaks to the terminal. The voice input device captures the user's speech, "Hello. How are you today?" and processes it as digital voice data.

[0492] Step 2:

[0493] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[0494] Step 3:

[0495] The server decodes the voice data received from the device and passes it to the ASR (Automatic Speech Recognition) engine for processing. The ASR engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?"

[0496] Step 4:

[0497] The server encodes the generated text data and transmits it over the network to the hearing-impaired user's device. The text data is packetized and transmitted according to the appropriate communication protocol.

[0498] Step 5:

[0499] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The hearing impaired person reads and understands this text.

[0500] Step 6:

[0501] A user (who is hearing impaired) uses the text input device of the device to input a reply, for example, "Hello. The weather is nice today and I'm in a good mood."

[0502] Step 7:

[0503] The terminal receives the input text data and transmits it to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[0504] Step 8:

[0505] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine for processing. The TTS engine analyzes the text data and generates the corresponding voice data: "Hello. The weather is nice today and I'm in a good mood."

[0506] Step 9:

[0507] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[0508] Step 10:

[0509] The device (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. The weather is nice today and I'm feeling great." using the device's speaker. A able-bodied person listens to and understands this voice.

[0510] These are the specific processing steps of the system, which allows for seamless communication between hearing-impaired and hearing-savvy people.

[0511] Example 1

[0512] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0513] In communication between hearing-impaired and hearing-savvy people, there is a need for an efficient and accurate system that can convert speech to text and vice versa in real time to achieve seamless dialogue. Conventional systems have insufficient conversion accuracy and speed, often causing stress and discomfort during actual dialogue. In addition, the need to use multiple different devices and applications makes operation cumbersome, and there have been few easy-to-use systems.

[0514] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0515] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, means for transmitting the converted text data, means for receiving text data entered by a user, means for converting the received text data into voice data, means for transmitting the converted voice data, means including an automatic speech recognition engine using a generative AI model, and means including a text-to-speech synthesis engine using a generative AI model, thereby enabling real-time and seamless dialogue between hearing-impaired people and hearing-normal people.

[0516] "Audio data" is data that represents an audio signal in digital form.

[0517] "Text data" is data that represents character information in digital form.

[0518] A "generative AI model" is an artificial intelligence model used to automatically generate and transform data.

[0519] An "automatic speech recognition engine" is a software component that analyzes speech data and converts it into corresponding text data.

[0520] A "text-to-speech engine" is a software component that analyzes text data and converts it into corresponding speech data.

[0521] "Network connectivity" is the infrastructure for communicating data between devices.

[0522] "User" refers to an individual who uses the system, and includes both hearing-impaired and hearing-nondisabled individuals.

[0523] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of the hearing-impaired person into text data, and converts the text data input by the hearing-impaired person into voice data, enabling natural dialogue in real time.

[0524] Overall system configuration

[0525] The system consists of the following main components:

[0526] Server: The generative AI model and its components, the automatic speech recognition engine (ASR) and the text-to-speech synthesis engine (TTS), run here, converting voice data and text data.

[0527] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0528] Users: Deaf and hearing impaired people who use the system.

[0529] Hardware and software used

[0530] Hardware: Various devices such as a microphone, speaker, display, keyboard, and network interface are required. Examples include smartphones, tablets, and PCs.

[0531] Software: Includes software components such as generative AI models, ASR engines (e.g., Google Cloud Speech-to-Text, Microsoft Azure Speech Services), and TTS engines (e.g., Google Cloud Text-to-Speech, AWS Polly).

[0532] Data processing and data calculation

[0533] The server receives the voice data sent from the device, analyzes it with an ASR engine, and converts it into text data. This text data is then sent back to the device and displayed on the screen. Conversely, the text data sent from the device is converted into voice data by the server with a TTS engine, and played back on the device. This series of processes enables seamless communication between hearing-impaired and hearing-normal people.

[0534] Specific examples

[0535] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0536] 1. Voice input from a healthy subject:

[0537] A healthy person speaks into the device: "Hello. How are you today?"

[0538] The terminal transmits the voice data to the server.

[0539] The server converts the voice data into text "Hello, how are you today?"

[0540] The text appears on the device of the hearing impaired person.

[0541] 2. Text Response for the Deaf:

[0542] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[0543] The terminal transmits the text data to the server.

[0544] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[0545] The device of the able-bodied person plays the audio.

[0546] Prompt Sentence Examples

[0547] "A hearing-impaired person inputs speech, which is converted into text and displayed to a hearing-impaired person. Please explain in detail the system that inputs this text, converts it back into speech, and plays it back to the hearing-impaired person."

[0548] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people. By utilizing generative AI models, highly accurate and real-time dialogue is possible, significantly removing communication barriers.

[0549] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0550] (Details of processing steps)

[0551] Step 1:

[0552] Voice input and capture for able-bodied people

[0553] Input: A healthy person speaks into the terminal.

[0554] Specific action: A healthy person says, "Hello. How are you today?"

[0555] Data processing and output: The microphone on the device captures the audio and stores it in a temporary buffer as digital audio data. This audio data is then sent to the server via the network.

[0556] Step 2:

[0557] Speech-to-text conversion

[0558] Input: Digital audio data sent from the device.

[0559] Specific operation: The server receives the audio data.

[0560] Data processing and output: An automatic speech recognition engine (ASR), part of the generative AI model on the server, analyzes the voice data and converts it into corresponding text data: "Hello. How are you today?" This text data is stored in a temporary database and sent to the hearing-impaired person's device.

[0561] Step 3:

[0562] Displaying Text

[0563] Input: Text data sent from the server.

[0564] Specific operation: The hearing-impaired person's device receives text data.

[0565] Data processing and output: Displaying text on the terminal display. For example, updating the text "Hello, how are you today?" in the display area.

[0566] Step 4:

[0567] Text entry for the hearing impaired

[0568] Input: A hearing-impaired person types text into a terminal.

[0569] Specific action: A hearing-impaired person types, "Hello. The weather is nice today and I'm feeling great."

[0570] Data processing and output: The input text data is temporarily stored on the terminal and sent to the server via the network.

[0571] Step 5:

[0572] Text to speech conversion

[0573] Input: Text data sent from the terminal.

[0574] Specific operation: The server receives the text data.

[0575] Data processing and output: A text-to-speech engine (TTS), part of the generative AI model on the server, analyzes the text data and converts it into corresponding voice data: "Hello. The weather is nice today and I'm feeling great." This voice data is stored in a temporary database and sent to the able-bodied user's device.

[0576] Step 6:

[0577] Playing audio

[0578] Input: Audio data sent from the server.

[0579] Specific operation: The device of the healthy person receives the voice data.

[0580] Data processing and output: The received voice data is played back through the device's speaker. For example, a healthy person hears a voice saying, "Hello. The weather is nice today and I'm feeling good."

[0581] By inputting, processing, and outputting data at each step, seamless real-time communication between hearing-impaired and hearing-speaking people is realized. The system uses speech recognition and speech synthesis technology that utilizes generative AI models to provide high accuracy and smooth dialogue.

[0582] (Application example 1)

[0583] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In modern society, there are limited ways for hearing-impaired and hearing-normal people to communicate effectively. In particular, in brick-and-mortar stores, hearing-impaired customers often have difficulty communicating with store staff, which can affect the quality of service. To solve this problem, a system is needed that allows both hearing-impaired and hearing-normal people to communicate naturally and efficiently.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0586] In this invention, the server includes a means for receiving voice data, a means for converting the received voice data into text data, and a means for transmitting the converted text data, thereby enabling two-way communication of text data and voice data between customers and store clerks in a physical store.

[0587] The "means for receiving audio data" is a device or function that acquires audio input from the outside as digital data.

[0588] The "means for converting received voice data into text data" refers to a device or function that analyzes voice data and converts it into corresponding text data.

[0589] The "means for transmitting converted text data" refers to a device or function that transmits the generated text data to another device or system.

[0590] The "means for receiving text data entered by a user" refers to a device or function that acquires text data entered by a user.

[0591] The "means for converting received text data into voice data" refers to a device or function that analyzes text data and converts it into corresponding voice data.

[0592] The "means for transmitting converted audio data" is a device or function that transmits the generated audio data to another device or system.

[0593] "Means for enabling two-way communication of text data and voice data between customers and store clerks in a physical store" refers to a device or function that allows customers and store clerks in a physical store to exchange information in text and voice format two-way.

[0594] This invention is a system that enables seamless communication between hearing-impaired people and hearing-normal people (store clerks) in brick-and-mortar stores. This system converts voice data into text data and text data into voice data, enabling natural dialogue in real time.

[0595] Overall system configuration

[0596] The system consists of the following main components:

[0597] Server: The server runs a speech recognition engine, a text-to-speech synthesis engine, and a data transmission / reception module, and converts voice data into text data.

[0598] Terminal: A device used by users (hearing impaired and hearing-able) that is capable of voice input / output, and text input / display, such as a smartphone.

[0599] Users: Deaf and hearing-impaired people (store clerks) who use the system.

[0600] Processing voice input

[0601] First, a person with normal hearing speaks into the device to input voice. For example, if a store clerk says, "Are you looking for something?", this voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0602] Speech-to-text conversion

[0603] The voice data received by the server is analyzed by a voice recognition engine, and the corresponding text data, "Are you looking for something?", is generated. This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[0604] Text entry for the hearing impaired

[0605] Next, the hearing-impaired person uses their own device to input a reply, for example, "Yes, I'm looking for shoes." This input text data is sent to the server by the device.

[0606] Text to speech conversion

[0607] The text data received by the server is analyzed by a text-to-speech synthesis engine, which generates the corresponding voice data, "Hello, I'm looking for shoes." This voice data is sent to the device of the able-bodied person and played back through a speaker.

[0608] Specific examples

[0609] For example, here is a concrete example of a conversation between a hearing person and a hearing-impaired person:

[0610] 1. Voice input from a healthy subject:

[0611] Store clerk: "Are you looking for something?"

[0612] The terminal transmits the voice data to the server.

[0613] The server converts the voice data into the text "Are you looking for something?"

[0614] The text appears on the device of the hearing impaired person.

[0615] 2. Text Response for the Deaf:

[0616] Deaf person: "Yes, I'm looking for my shoes."

[0617] The terminal transmits the text data to the server.

[0618] The server converts the text data into speech, "Yes, I'm looking for shoes."

[0619] The clerk's terminal plays the audio.

[0620] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people in brick-and-mortar stores. This invention significantly improves the environment in which hearing-impaired people receive services in brick-and-mortar stores, enabling both parties to communicate efficiently.

[0621] Prompt Sentence Examples

[0622] "Hello. How are you today? Is there anything I can help you with?"

[0623] The above configuration provides an environment in which hearing-impaired and hearing-normal people can communicate smoothly in physical stores.

[0624] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0625] Step 1:

[0626] Voice input is performed when a person with hearing loss speaks into the device. The device captures this voice input as digital voice data. The input is what the person with hearing loss said (voice), and the output is digital voice data. This digital voice data is temporarily stored in the device.

[0627] Step 2:

[0628] The terminal transmits the captured audio data to the server via the network. The input is the captured digital audio data, and the output is the audio data transmitted to the server. This is where the data transmission module operates.

[0629] Step 3:

[0630] The speech recognition engine analyzes the voice data received by the server and converts it into corresponding text data. The input is the received digital voice data and the output is the corresponding text data. The speech recognition engine uses a generative AI model to analyze the voice and generate accurate text.

[0631] Step 4:

[0632] The server sends the generated text data to the terminal of the hearing impaired person. The input is the converted text data, and the output is the text data sent to the terminal of the hearing impaired person. Here, the data sending module is operated again.

[0633] Step 5:

[0634] The hearing impaired person's device displays the received text data on the screen. The input is the received text data, and the output is the text displayed on the display screen. The display module of the device operates to provide the text visually.

[0635] Step 6:

[0636] A hearing-impaired person types a text reply into their device. The input is text data entered by the hearing-impaired person, and the output is text data stored on the device. This text is entered using the device's input device (keyboard or touch screen).

[0637] Step 7:

[0638] The terminal sends the input text data to the server. The input is the text data entered by the hearing impaired person, and the output is the text data sent to the server. The terminal transfers the data via the network.

[0639] Step 8:

[0640] The text data received by the server is analyzed by the text-to-speech engine and converted into corresponding voice data. The input is the received text data and the output is the generated voice data. The text-to-speech engine uses a generative AI model to generate natural-sounding voices.

[0641] Step 9:

[0642] The server sends the generated voice data to the device of the able-bodied person. The input is the generated voice data, and the output is the voice data sent to the device of the able-bodied person. Here too, the data transmission module operates.

[0643] Step 10:

[0644] The device of the able-bodied person plays the received voice data. The input is the received voice data, and the output is the voice to be played. The speaker module of the device operates and provides the voice to the able-bodied person.

[0645] These are the specific processing steps in this system. At each step, data is processed and calculated, ultimately achieving natural conversation between hearing-impaired and hearing-normal people.

[0646] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0647] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts the voice input of hearing-impaired people into text data and the text data input of hearing-impaired people into voice data, but also analyzes and conveys emotional information, enabling more natural and rich communication. Here, the detailed operation of this system is explained in natural language.

[0648] Overall system configuration

[0649] The system consists of the following main components:

[0650] Server: The generation AI, chat generation AI, and emotion engine run, converting voice data and text data.

[0651] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0652] Users: Deaf and hearing impaired people who use the system.

[0653] Voice input and emotion recognition

[0654] First, a person with normal hearing speaks into the device to input voice. "Hello. How are you today?" the person says. This voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0655] The voice data received by the server is analyzed by an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. The emotional information recognized by the emotion engine is added as a tag to the converted text data. For example, it may be in the form of "Hello. How are you today? (Joy)."

[0656] Displaying text data and conveying emotions

[0657] The emotion-tagged text data generated by the server is sent over the network to the hearing-impaired person's device. The device decodes the received text data and displays "Hello. How are you today?" on the screen, along with visually displaying emotion information using icons and colors that indicate the emotion tag. This allows the hearing-impaired person to understand the emotions of people without hearing.

[0658] Text input and emotion communication for the hearing impaired

[0659] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This input text data is sent to the server along with the emotion information.

[0660] Text-to-speech and emotional response

[0661] The text data received by the server is passed to a TTS (Text to Speech) engine along with emotional information. The TTS engine analyzes the text data and generates voice data, "Hello. It's nice weather today and I'm feeling great," based on the emotional information. This voice data is given an intonation that corresponds to the emotional information. For example, if the emotion tag is "joy," a lively and cheerful voice is generated.

[0662] Specific examples

[0663] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0664] 1. Speech input and emotion recognition for healthy subjects:

[0665] A healthy person speaks into the device: "Hello. How are you today?" (joy)

[0666] The terminal transmits the voice data to the server.

[0667] The server analyzes the voice data and recognizes the emotion as "joy."

[0668] The voice data is converted into text "Hello. How are you today?" and an emotion tag (joy) is assigned.

[0669] Text and emotional information is sent to the device of a hearing-impaired person and displayed visually.

[0670] 2. Text replies and emotional reflection for the deaf:

[0671] A deaf person types text into a device: "Hello. The weather is nice today and I'm feeling great." (Joy)

[0672] The device sends text data and emotion information to the server.

[0673] The server analyzes the text data and generates a voice message based on the emotional information: "Hello. The weather is nice today and I'm in a good mood."

[0674] The emotion of "joy" is reflected in the audio data and played back on a device belonging to a healthy person.

[0675] This system will enable more natural and rich communication between hearing-impaired and hearing-suffering people, incorporating emotional information. The introduction of an emotion engine will enable dialogue to go beyond simply exchanging information, and involve the transmission of emotions, deepening mutual understanding.

[0676] The processing flow will be explained below.

[0677] Step 1:

[0678] A user (with normal hearing) speaks to the device. The voice input device captures the user's speech and processes it as digital voice data.

[0679] Step 2:

[0680] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[0681] Step 3:

[0682] The server decodes the voice data received from the device and passes it to the emotion engine, which analyzes the voice data and recognizes the user's emotion (e.g., "happiness," "sadness," "anger," etc.).

[0683] Step 4:

[0684] The server receives the emotion information from the emotion engine and passes the voice data to the ASR (Automatic Speech Recognition) engine, which analyzes the voice data and generates corresponding text data (e.g., "Hello. How are you today?").

[0685] Step 5:

[0686] The server tags the generated text data with emotional information recognized by the emotion engine (e.g., "Hello. How are you today? (Joy)").

[0687] Step 6:

[0688] The server encodes the text data with emotion tags and sends it over the network to the hearing-impaired person's device. The text data is packetized and sent according to the appropriate communication protocol.

[0689] Step 7:

[0690] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The emotion tag (joy) is visually displayed using icons and colors.

[0691] Step 8:

[0692] A user (who is hearing impaired) uses the device's text input device to type a reply (e.g., "Hello. I'm in a good mood today.") and also types an emotion tag (joy).

[0693] Step 9:

[0694] The device receives the input text data and emotion tag, and transmits them to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[0695] Step 10:

[0696] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine based on the emotion tag. The TTS engine analyzes the text data and generates corresponding voice data (e.g., "Hello. I'm in a good mood today because the weather is nice.").

[0697] Step 11:

[0698] The server adds emotional intonation to the voice data based on the emotion tag (joy), generating voice data that reflects the emotion.

[0699] Step 12:

[0700] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[0701] Step 13:

[0702] The terminal (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. It's nice weather today and I'm feeling good" using a speaker. The voice reflects emotional intonation, and able-bodied people can understand this voice.

[0703] These are the specific processing steps of a system that combines an emotion engine. This processing enables more natural and richer dialogue that includes emotional information.

[0704] Example 2

[0705] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0706] Conventional communication systems have difficulty in achieving natural dialogue, including emotional content, between hearing-impaired and hearing-challenged people. Simply converting the speech input of hearing-impaired people into text and the text input of hearing-impaired people into speech results in the loss of emotional nuances and the resulting dialogue becoming monotonous. This drawback is particularly serious in communication where sharing emotions is important, making it difficult for people to understand each other's emotions.

[0707] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing emotional information from the received voice data, means for converting the voice data with added emotional information into text data, means for transmitting the converted text data and emotional information, means for receiving text data entered by the user together with the emotional information, means for converting the received text data and emotional information into voice data, and means for transmitting the converted voice data and emotional information. This enables more natural and rich communication, including emotion, between hearing-impaired people and those with hearing disabilities.

[0708] The "means for receiving voice data" refers to a mechanism for inputting voice data generated from an input device of a healthy person into a server.

[0709] "Means for analyzing emotion information" refers to a mechanism for identifying the user's emotion from received voice data using an emotion engine and extracting that information.

[0710] "Means for converting voice data into text data" refers to a mechanism for converting received and analyzed voice data into text information via an ASR (automatic speech recognition) engine.

[0711] "Means for transmitting text data to which emotional information has been added" refers to a mechanism for adding emotional information to converted text data and transmitting it to another terminal via a network.

[0712] The "means for receiving text data entered by a user together with emotion information" refers to a mechanism for capturing the character information entered by a user into a terminal and the selected emotion tag into a server.

[0713] "Means for converting text data and emotional information into voice data" refers to a mechanism for converting received text data and emotional information into voice data via a TTS (Text to Speech Synthesis) engine.

[0714] The "means for transmitting the converted voice data and emotional information" refers to a mechanism for transmitting the generated voice data and its emotional information to another terminal via a network.

[0715] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts voice data into text data but also analyzes and transmits emotional information, enabling more natural and rich communication.

[0716] Voice input and emotion recognition

[0717] When a user (a person with normal hearing) speaks into the device, their voice is captured using a microphone and processed as digital voice data. For example, a person with normal hearing might say, "Hello. How are you today?" This voice data is sent to a server via a network.

[0718] Audio data processing on the server

[0719] The server analyzes the received voice data using an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. Emotional information is added to the converted text data as a tag. For example, it might look like "Hello. How are you today? (Joy)". This generates data that includes text data and emotional information.

[0720] Displaying text data and conveying emotions

[0721] The device receives the emotion-tagged text data sent from the server and displays it on the screen, while visually displaying emotion information using icons and colors that indicate the emotion tag. This allows hearing-impaired people to understand the emotions of hearing-impaired people.

[0722] Text input and emotion communication for the hearing impaired

[0723] Next, the hearing-impaired person uses their own device to input a reply text. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This text data and emotional information are sent to the server via the network.

[0724] Text data processing on the server

[0725] The text data received by the server is passed to a TTS (Text to Speech) engine. The TTS engine analyzes the text data and generates voice data based on the emotional information. For example, if an emotional tag (joy) is attached, a lively and cheerful voice saying "Hello. The weather is nice today and I'm in a good mood" is generated.

[0726] Playing audio data

[0727] The generated voice data is sent to the device of the hearing-impaired person and played back, allowing the hearing-impaired person to understand the emotions of the hearing-impaired person.

[0728] Specific examples

[0729] Here are some examples of specific prompts:

[0730] 1. "Turn the following speech data into text and analyze the user's sentiment: Hello. How are you today?"

[0731] 2. "Generate speech from emotion-tagged text data: Hello. The weather is nice today and I'm feeling great. (Joy)"

[0732] Hardware and software used

[0733] Devices: Smartphones, tablets, PCs, etc.

[0734] Server: Generation AI, chat generation AI, emotion engine, ASR engine, TTS engine

[0735] Emotion Recognition Software: Emotion Engine

[0736] Automatic Speech Recognition Software: ASR Engine

[0737] Text to Speech Software: TTS Engine

[0738] By using this system, communication between hearing-impaired and hearing-challenged people can take place in a way that includes emotional information, enabling a more natural and deeper mutual understanding.

[0739] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0740] Step 1:

[0741] Device:

[0742] A user (healthy person) speaks into the device. Example: "Hello. How are you today?"

[0743] Input: Normal human speech

[0744] How it works: The device's microphone captures audio and converts it into audio data.

[0745] Output: Digital audio data

[0746] Step 2:

[0747] Device:

[0748] Input: Captured digital audio data

[0749] What it does: The device sends this digital audio data to the server.

[0750] Output: Audio data sent to the server

[0751] Step 3:

[0752] server:

[0753] Input: Transmitted audio data

[0754] How it works: The server analyzes this voice data through an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger).

[0755] Output: Audio data with emotional information added

[0756] Step 4:

[0757] server:

[0758] Input: Audio data with emotional information

[0759] How it works: This data is converted into text using an ASR (automatic speech recognition) engine.

[0760] Output: Emotion-tagged text data (e.g., "Hello. How are you today? (Joy)")

[0761] Step 5:

[0762] server:

[0763] Input: Emotion-tagged text data

[0764] How it works: The server sends the generated emotion-tagged text data to the hearing-impaired person's device via the network.

[0765] Output: Text data and emotional information sent to the hearing-impaired person's device

[0766] Step 6:

[0767] Terminal (hearing impaired):

[0768] Input: Emotion-tagged text data

[0769] How it works: The device displays the received text data on the screen and visually represents the emotional information using icons and colors.

[0770] Output: Text and emotion information displayed on the screen

[0771] Step 7:

[0772] User (hearing impaired):

[0773] Input: Terminal input field

[0774] Action: A deaf person types a reply. Example: "Hello. The weather is nice today and I'm feeling great."

[0775] Output: Input text data

[0776] Step 8:

[0777] Terminal (hearing impaired):

[0778] Input: Text data entered by a hearing-impaired person

[0779] What it does: Sends the text and the selected emotion tag to the server.

[0780] Output: Text data and emotion information sent to the server

[0781] Step 9:

[0782] server:

[0783] Input: Received text data and emotion information

[0784] How it works: The server passes this to a TTS (Text to Speech) engine, which converts the received text data into voice data. At that time, it adds an appropriate tone to the voice based on the emotion tag. For example, for the (joy) tag, a lively and cheerful tone

[0785] Output: Audio data reflecting emotional information

[0786] Step 10:

[0787] server:

[0788] Input: Voice data reflecting emotional information

[0789] Operation: The server transmits the generated voice data to the device of the healthy person via the network.

[0790] Output: Audio data sent to the healthy subject's device

[0791] Step 11:

[0792] Device (healthy person):

[0793] Input: Transmitted audio data

[0794] Action: Plays back the audio data and has a normal subject listen to it. Example: "Hello. The weather is nice today and I'm feeling great."

[0795] Output: Played audio data

[0796] Through these steps, natural communication, including emotional content, is realized between hearing-impaired and hearing-challenged people.

[0797] (Application example 2)

[0798] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] It is difficult to achieve smooth and natural communication between hearing-impaired and hearing-savvy people. Conventional systems only convert voice data to text data, and lack emotional exchange in communication. This leads to insufficient communication and a decline in the quality of communication. Especially in brick-and-mortar stores, there is a strong demand for communication that includes emotional information, as this has a significant impact on customer experience and service quality.

[0800] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, emotion analysis means for analyzing the user's emotion and generating emotion information, means for assigning the generated emotion information to the text data, means for visualizing the assigned emotion information, and means for reflecting the assigned emotion information in the voice data. This enables natural and rich communication including emotion information between hearing-impaired and hearing-normal people.

[0801] "Means for receiving audio data" refers to a device or method that receives an external audio signal with low loss and converts it into a format suitable for internal processing.

[0802] A "means for converting received voice data into text data" is a device or method that uses voice recognition technology to convert received voice signals into a corresponding text format.

[0803] The "means for transmitting converted text data" refers to a device or method for transmitting the converted text data to another device or system.

[0804] The "means for receiving text data entered by a user" refers to a device or method that receives text data entered by a user and converts it into a format suitable for internal processing.

[0805] A "means for converting received text data into speech data" is a device or method that uses text-to-speech technology to convert text data into a corresponding speech format.

[0806] The "means for transmitting converted audio data" refers to a device or method for transmitting the converted audio data to another device or system.

[0807] "Emotion analysis means" refers to a device or method that reads the user's emotions from input voice or text data and generates emotion information.

[0808] The "means for adding generated emotion information to text data" refers to a device or method for adding emotion information obtained by emotion analysis to text and reinforcing the text data.

[0809] The "means for visualizing assigned emotional information" is a device or method for visualizing analyzed emotional information and presenting it to the user.

[0810] The "means for reflecting the added emotional information in the voice data" refers to a device or method for incorporating the generated emotional information into the voice synthesis process and expressing it as voice.

[0811] Overall system configuration

[0812] This system aims to realize seamless communication between hearing-impaired and hearing-suffering people. The system is broadly composed of the following elements:

[0813] Server: The emotion analysis, speech recognition, and text-to-speech synthesis engines run on a high-performance cloud server.

[0814] Terminal: A device capable of voice input / output and text display, such as a smartphone or smart glasses.

[0815] Users: Deaf and hearing impaired people who use the system.

[0816] Hardware / Software used

[0817] Server: A cloud server such as AWS EC2 or Google Cloud.

[0818] Emotion engine: Microsoft Azure Emotion API.

[0819] Speech recognition engine (ASR): Google Cloud Speech-to-Text.

[0820] Text-to-speech engine (TTS): Amazon Polly.

[0821] Program processing explanation

[0822] Voice input and emotion recognition

[0823] 1. Terminal: Voice input is performed by a healthy person speaking into the terminal.

[0824] 2. Server: The audio data captured on the device is sent to the server.

[0825] 3. Server (emotion analysis means): The received voice data is analyzed by the emotion engine, and emotional information such as joy, sadness, and anger is generated.

[0826] 4. Server (voice recognition means): Voice data is converted into text data by the ASR engine.

[0827] 5. Server (means for adding emotional information): Emotional information is added to text data as tags. For example, "Hello. How are you today? (Joy)".

[0828] Displaying text data and conveying emotions

[0829] 6. Server: The generated emotion-tagged text data is sent to the hearing-impaired person's device.

[0830] 7. Terminal (visualization means): The text data is decoded and displayed on the screen, and emotion tags are visually displayed using icons and colors.

[0831] Text input and emotion communication for the hearing impaired

[0832] 8. Device: A hearing impaired person uses a device to input text, for example, "Hello. The weather is nice today and I feel good. (Joy)."

[0833] 9. Server: The input text data is sent to the server along with the emotion information.

[0834] Text-to-speech and emotional response

[0835] 10. Server (text-to-speech synthesis means): The received text data is passed to the TTS engine, which analyzes the text data and generates voice data.

[0836] 11. Server (Means for Reflecting Emotional Information): The server adds intonation to the voice data based on the emotional information. For example, if the voice data is tagged with "joy," it generates a voice with a lively and cheerful tone.

[0837] 12. Terminal: The generated voice data is played on the terminal of the able-bodied person.

[0838] Examples of concrete examples and prompts

[0839] Here is a concrete example of the dialogue:

[0840] 1. Healthy person: "Hello. How are you today?" (joy).

[0841] 2. Hearing impaired person: "Hello. The weather is nice today and I feel great." (joy).

[0842] Example prompt for a generative AI model:

[0843] "The store clerk greeted: Hello, is there anything you're looking for?"

[0844] "Customer enters through smart glasses: I'm not looking for anything in particular today."

[0845] "Convert these messages into text and audio with appropriate emotional information."

[0846] This system enables natural and rich communication, including emotional information, between hearing-impaired and hearing-savvy people. Specifically, it is expected to improve the customer experience and service quality in brick-and-mortar stores.

[0847] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0848] Step 1:

[0849] Device: Voice input is performed by a non-disabled person speaking into the device. The device's built-in microphone captures the voice signal and stores it as digitally formatted voice data. This voice data is sent to a server for further processing. The input here is the voice spoken by a non-disabled person, and the output is digitally formatted voice data.

[0850] Step 2:

[0851] Server: Receives the voice data sent from the device. The server checks the format of the voice data and converts it to the appropriate format if necessary. At this stage, data pre-processing is performed, such as noise filtering. The input is voice data, and the output is pre-processed voice data.

[0852] Step 3:

[0853] Server (emotion analysis means): Analyzes the received voice data using an emotion engine to generate emotional information for healthy individuals. In the analysis step, emotions are estimated based on the pitch, tone, and speed of the voice. The input is processed voice data, and the output is metadata containing emotional information.

[0854] Step 4:

[0855] Server (voice recognition means): After emotion analysis, the voice data is converted into text data by the ASR engine. The voice recognition engine analyzes the voice waveform and converts it into the corresponding text format. The input here is the voice data that has been emotion analyzed, and the output is the converted text data.

[0856] Step 5:

[0857] Server (emotion information adding means): Adds the generated emotion information to the converted text data as a tag. Specifically, emotion information is added to the end of a sentence in the text. For example, "Hello. How are you today? (Joy)". The input is text data and emotion metadata, and the output is emotion-tagged text data.

[0858] Step 6:

[0859] Server: Sends emotion-tagged text data to the device of the hearing-impaired person. Data transfer takes place here, and stable communication is ensured using network protocols (e.g., HTTP, WebSocket). The input is emotion-tagged text data, and the output is the text data received by the device of the hearing-impaired person.

[0860] Step 7:

[0861] Terminal (Visualization Means): The terminal of the hearing-impaired person decodes the received text data and displays it on the screen. At the same time, visual feedback (e.g., icons or colors) based on the emotion tag is also displayed. The input is emotion-tagged text data, and the output is the displayed text and emotion information.

[0862] Step 8:

[0863] User: A hearing-impaired person uses the device to input a text message. For example, "Hello. The weather is nice today and I feel great. (Joy)." The device receives the user's input and stores it as text data. The input is the text entered by the user, and the output is text data.

[0864] Step 9:

[0865] Terminal: Sends input text data to the server. The terminal's communication module sends the text data to the server, and the data is encrypted if necessary. The input is text data, and the output is the text data sent to the server.

[0866] Step 10:

[0867] Server (Text-to-Speech Synthesis Means): The received text data is converted into voice data by the TTS engine. The TTS engine analyzes the text and generates natural-sounding voice. The input is text data, and the output is the generated voice data.

[0868] Step 11:

[0869] Server (emotional information reflection means): Reflects emotional information in the generated voice data. Specifically, it adjusts the tone and pitch of the voice based on the emotion tag. The input is voice data and emotional information, and the output is voice data that reflects the emotional information.

[0870] Step 12:

[0871] Terminal: The audio data reflecting emotional information is played on the terminal of the hearing-impaired person. Here, the hearing-impaired person can receive a message from the hearing-impaired person by voice. The input is the audio data reflecting emotional information, and the output is the audio played from the terminal.

[0872] In this way, the system can realize natural communication including emotional information between hearing-impaired and hearing-normal people.

[0873] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0874] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0875] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0876] [Third embodiment]

[0877] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0878] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0879] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0880] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0881] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0882] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0883] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0884] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0885] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0886] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0887] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0888] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0889] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of hearing-impaired people into text data, and converts the text data input by hearing-impaired people into voice data, enabling natural dialogue in real time. Here, the detailed operation of this system is explained in natural language.

[0890] Overall system configuration

[0891] The system consists of the following main components:

[0892] Server: The generation AI and chat generation AI run, converting voice data and text data.

[0893] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0894] Users: Deaf and hearing impaired people who use the system.

[0895] Processing voice input

[0896] First, a person with normal hearing speaks into the device to input voice. When a person with normal hearing speaks, "Hello. How are you today?", the voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[0897] Speech-to-text conversion

[0898] The voice data received by the server is processed by the ASR (Automatic Speech Recognition) engine, a component of the generative AI. This engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?" This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[0899] Text entry for the hearing impaired

[0900] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling great." This input text data is sent to the server by the device.

[0901] Text to speech conversion

[0902] The text data received by the server is processed by a TTS (Text to Speech) engine, a component of the chat generation AI. This engine analyzes the text data and generates the corresponding voice data, "Hello. The weather is nice today and I'm in a good mood." This generated voice data is sent to the device of the able-bodied person and played back through the speaker.

[0903] Specific examples

[0904] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0905] 1. Voice input from a healthy subject:

[0906] A healthy person speaks into the device: "Hello. How are you today?"

[0907] The terminal transmits the voice data to the server.

[0908] The server converts the voice data into text "Hello, how are you today?"

[0909] The text appears on the device of the hearing impaired person.

[0910] 2. Text Response for the Deaf:

[0911] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[0912] The terminal transmits the text data to the server.

[0913] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[0914] The device of the able-bodied person plays the audio.

[0915] This system will enable natural and seamless communication between hearing-impaired and hearing-suffering people. This invention will significantly improve the communication environment for hearing-impaired people, enabling both parties to communicate efficiently.

[0916] The processing flow will be explained below.

[0917] Step 1:

[0918] A user (with normal hearing) speaks to the terminal. The voice input device captures the user's speech, "Hello. How are you today?" and processes it as digital voice data.

[0919] Step 2:

[0920] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[0921] Step 3:

[0922] The server decodes the voice data received from the device and passes it to the ASR (Automatic Speech Recognition) engine for processing. The ASR engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?"

[0923] Step 4:

[0924] The server encodes the generated text data and transmits it over the network to the hearing-impaired user's device. The text data is packetized and transmitted according to the appropriate communication protocol.

[0925] Step 5:

[0926] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The hearing impaired person reads and understands this text.

[0927] Step 6:

[0928] A user (who is hearing impaired) uses the text input device of the device to input a reply, for example, "Hello. The weather is nice today and I'm in a good mood."

[0929] Step 7:

[0930] The terminal receives the input text data and transmits it to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[0931] Step 8:

[0932] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine for processing. The TTS engine analyzes the text data and generates the corresponding voice data: "Hello. The weather is nice today and I'm in a good mood."

[0933] Step 9:

[0934] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[0935] Step 10:

[0936] The device (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. The weather is nice today and I'm feeling great." using the device's speaker. A able-bodied person listens to and understands this voice.

[0937] These are the specific processing steps of the system, which allows for seamless communication between hearing-impaired and hearing-savvy people.

[0938] Example 1

[0939] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0940] In communication between hearing-impaired and hearing-savvy people, there is a need for an efficient and accurate system that can convert speech to text and vice versa in real time to achieve seamless dialogue. Conventional systems have insufficient conversion accuracy and speed, often causing stress and discomfort during actual dialogue. In addition, the need to use multiple different devices and applications makes operation cumbersome, and there have been few easy-to-use systems.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0942] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, means for transmitting the converted text data, means for receiving text data entered by a user, means for converting the received text data into voice data, means for transmitting the converted voice data, means including an automatic speech recognition engine using a generative AI model, and means including a text-to-speech synthesis engine using a generative AI model, thereby enabling real-time and seamless dialogue between hearing-impaired people and hearing-normal people.

[0943] "Audio data" is data that represents an audio signal in digital form.

[0944] "Text data" is data that represents character information in digital form.

[0945] A "generative AI model" is an artificial intelligence model used to automatically generate and transform data.

[0946] An "automatic speech recognition engine" is a software component that analyzes speech data and converts it into corresponding text data.

[0947] A "text-to-speech engine" is a software component that analyzes text data and converts it into corresponding speech data.

[0948] "Network connectivity" is the infrastructure for communicating data between devices.

[0949] "User" refers to an individual who uses the system, and includes both hearing-impaired and hearing-nondisabled individuals.

[0950] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of the hearing-impaired person into text data, and converts the text data input by the hearing-impaired person into voice data, enabling natural dialogue in real time.

[0951] Overall system configuration

[0952] The system consists of the following main components:

[0953] Server: The generative AI model and its components, the automatic speech recognition engine (ASR) and the text-to-speech synthesis engine (TTS), run here, converting voice data and text data.

[0954] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[0955] Users: Deaf and hearing impaired people who use the system.

[0956] Hardware and software used

[0957] Hardware: Various devices such as a microphone, speaker, display, keyboard, and network interface are required. Examples include smartphones, tablets, and PCs.

[0958] Software: Includes software components such as generative AI models, ASR engines (e.g., Google Cloud Speech-to-Text, Microsoft Azure Speech Services), and TTS engines (e.g., Google Cloud Text-to-Speech, AWS Polly).

[0959] Data processing and data calculation

[0960] The server receives the voice data sent from the device, analyzes it with an ASR engine, and converts it into text data. This text data is then sent back to the device and displayed on the screen. Conversely, the text data sent from the device is converted into voice data by the server with a TTS engine, and played back on the device. This series of processes enables seamless communication between hearing-impaired and hearing-normal people.

[0961] Specific examples

[0962] Here is a concrete example of a conversation between a hearing person and a deaf person:

[0963] 1. Voice input from a healthy subject:

[0964] A healthy person speaks into the device: "Hello. How are you today?"

[0965] The terminal transmits the voice data to the server.

[0966] The server converts the voice data into text "Hello, how are you today?"

[0967] The text appears on the device of the hearing impaired person.

[0968] 2. Text Response for the Deaf:

[0969] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[0970] The terminal transmits the text data to the server.

[0971] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[0972] The device of the able-bodied person plays the audio.

[0973] Prompt Sentence Examples

[0974] "A hearing-impaired person inputs speech, which is converted into text and displayed to a hearing-impaired person. Please explain in detail the system that inputs this text, converts it back into speech, and plays it back to the hearing-impaired person."

[0975] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people. By utilizing generative AI models, highly accurate and real-time dialogue is possible, significantly removing communication barriers.

[0976] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0977] (Details of processing steps)

[0978] Step 1:

[0979] Voice input and capture for able-bodied people

[0980] Input: A healthy person speaks into the terminal.

[0981] Specific action: A healthy person says, "Hello. How are you today?"

[0982] Data processing and output: The microphone on the device captures the audio and stores it in a temporary buffer as digital audio data. This audio data is then sent to the server via the network.

[0983] Step 2:

[0984] Speech-to-text conversion

[0985] Input: Digital audio data sent from the device.

[0986] Specific operation: The server receives the audio data.

[0987] Data processing and output: An automatic speech recognition engine (ASR), part of the generative AI model on the server, analyzes the voice data and converts it into corresponding text data: "Hello. How are you today?" This text data is stored in a temporary database and sent to the hearing-impaired person's device.

[0988] Step 3:

[0989] Displaying Text

[0990] Input: Text data sent from the server.

[0991] Specific operation: The hearing-impaired person's device receives text data.

[0992] Data processing and output: Displaying text on the terminal display. For example, updating the text "Hello, how are you today?" in the display area.

[0993] Step 4:

[0994] Text entry for the hearing impaired

[0995] Input: A hearing-impaired person types text into a terminal.

[0996] Specific action: A hearing-impaired person types, "Hello. The weather is nice today and I'm feeling great."

[0997] Data processing and output: The input text data is temporarily stored on the terminal and sent to the server via the network.

[0998] Step 5:

[0999] Text to speech conversion

[1000] Input: Text data sent from the terminal.

[1001] Specific operation: The server receives the text data.

[1002] Data processing and output: A text-to-speech engine (TTS), part of the generative AI model on the server, analyzes the text data and converts it into corresponding voice data: "Hello. The weather is nice today and I'm feeling great." This voice data is stored in a temporary database and sent to the able-bodied user's device.

[1003] Step 6:

[1004] Playing audio

[1005] Input: Audio data sent from the server.

[1006] Specific operation: The device of the healthy person receives the voice data.

[1007] Data processing and output: The received voice data is played back through the device's speaker. For example, a healthy person hears a voice saying, "Hello. The weather is nice today and I'm feeling good."

[1008] By inputting, processing, and outputting data at each step, seamless real-time communication between hearing-impaired and hearing-speaking people is realized. The system uses speech recognition and speech synthesis technology that utilizes generative AI models to provide high accuracy and smooth dialogue.

[1009] (Application example 1)

[1010] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1011] In modern society, there are limited ways for hearing-impaired and hearing-normal people to communicate effectively. In particular, in brick-and-mortar stores, hearing-impaired customers often have difficulty communicating with store staff, which can affect the quality of service. To solve this problem, a system is needed that allows both hearing-impaired and hearing-normal people to communicate naturally and efficiently.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1013] In this invention, the server includes a means for receiving voice data, a means for converting the received voice data into text data, and a means for transmitting the converted text data, thereby enabling two-way communication of text data and voice data between customers and store clerks in a physical store.

[1014] The "means for receiving audio data" is a device or function that acquires audio input from the outside as digital data.

[1015] The "means for converting received voice data into text data" refers to a device or function that analyzes voice data and converts it into corresponding text data.

[1016] The "means for transmitting converted text data" refers to a device or function that transmits the generated text data to another device or system.

[1017] The "means for receiving text data entered by a user" refers to a device or function that acquires text data entered by a user.

[1018] The "means for converting received text data into voice data" refers to a device or function that analyzes text data and converts it into corresponding voice data.

[1019] The "means for transmitting converted audio data" is a device or function that transmits the generated audio data to another device or system.

[1020] "Means for enabling two-way communication of text data and voice data between customers and store clerks in a physical store" refers to a device or function that allows customers and store clerks in a physical store to exchange information in text and voice format two-way.

[1021] This invention is a system that enables seamless communication between hearing-impaired people and hearing-normal people (store clerks) in brick-and-mortar stores. This system converts voice data into text data and text data into voice data, enabling natural dialogue in real time.

[1022] Overall system configuration

[1023] The system consists of the following main components:

[1024] Server: The server runs a speech recognition engine, a text-to-speech synthesis engine, and a data transmission / reception module, and converts voice data into text data.

[1025] Terminal: A device used by users (hearing impaired and hearing-able) that is capable of voice input / output, and text input / display, such as a smartphone.

[1026] Users: Deaf and hearing-impaired people (store clerks) who use the system.

[1027] Processing voice input

[1028] First, a person with normal hearing speaks into the device to input voice. For example, if a store clerk says, "Are you looking for something?", this voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[1029] Speech-to-text conversion

[1030] The voice data received by the server is analyzed by a voice recognition engine, and the corresponding text data, "Are you looking for something?", is generated. This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[1031] Text entry for the hearing impaired

[1032] Next, the hearing-impaired person uses their own device to input a reply, for example, "Yes, I'm looking for shoes." This input text data is sent to the server by the device.

[1033] Text to speech conversion

[1034] The text data received by the server is analyzed by a text-to-speech synthesis engine, which generates the corresponding voice data, "Hello, I'm looking for shoes." This voice data is sent to the device of the able-bodied person and played back through a speaker.

[1035] Specific examples

[1036] For example, here is a concrete example of a conversation between a hearing person and a hearing-impaired person:

[1037] 1. Voice input from a healthy subject:

[1038] Store clerk: "Are you looking for something?"

[1039] The terminal transmits the voice data to the server.

[1040] The server converts the voice data into the text "Are you looking for something?"

[1041] The text appears on the device of the hearing impaired person.

[1042] 2. Text Response for the Deaf:

[1043] Deaf person: "Yes, I'm looking for my shoes."

[1044] The terminal transmits the text data to the server.

[1045] The server converts the text data into speech, "Yes, I'm looking for shoes."

[1046] The clerk's terminal plays the audio.

[1047] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people in brick-and-mortar stores. This invention significantly improves the environment in which hearing-impaired people receive services in brick-and-mortar stores, enabling both parties to communicate efficiently.

[1048] Prompt Sentence Examples

[1049] "Hello. How are you today? Is there anything I can help you with?"

[1050] The above configuration provides an environment in which hearing-impaired and hearing-normal people can communicate smoothly in physical stores.

[1051] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1052] Step 1:

[1053] Voice input is performed when a person with hearing loss speaks into the device. The device captures this voice input as digital voice data. The input is what the person with hearing loss said (voice), and the output is digital voice data. This digital voice data is temporarily stored in the device.

[1054] Step 2:

[1055] The terminal transmits the captured audio data to the server via the network. The input is the captured digital audio data, and the output is the audio data transmitted to the server. This is where the data transmission module operates.

[1056] Step 3:

[1057] The speech recognition engine analyzes the voice data received by the server and converts it into corresponding text data. The input is the received digital voice data and the output is the corresponding text data. The speech recognition engine uses a generative AI model to analyze the voice and generate accurate text.

[1058] Step 4:

[1059] The server sends the generated text data to the terminal of the hearing impaired person. The input is the converted text data, and the output is the text data sent to the terminal of the hearing impaired person. Here, the data sending module is operated again.

[1060] Step 5:

[1061] The hearing impaired person's device displays the received text data on the screen. The input is the received text data, and the output is the text displayed on the display screen. The display module of the device operates to provide the text visually.

[1062] Step 6:

[1063] A hearing-impaired person types a text reply into their device. The input is text data entered by the hearing-impaired person, and the output is text data stored on the device. This text is entered using the device's input device (keyboard or touch screen).

[1064] Step 7:

[1065] The terminal sends the input text data to the server. The input is the text data entered by the hearing impaired person, and the output is the text data sent to the server. The terminal transfers the data via the network.

[1066] Step 8:

[1067] The text data received by the server is analyzed by the text-to-speech engine and converted into corresponding voice data. The input is the received text data and the output is the generated voice data. The text-to-speech engine uses a generative AI model to generate natural-sounding voices.

[1068] Step 9:

[1069] The server sends the generated voice data to the device of the able-bodied person. The input is the generated voice data, and the output is the voice data sent to the device of the able-bodied person. Here too, the data transmission module operates.

[1070] Step 10:

[1071] The device of the able-bodied person plays the received voice data. The input is the received voice data, and the output is the voice to be played. The speaker module of the device operates and provides the voice to the able-bodied person.

[1072] These are the specific processing steps in this system. At each step, data is processed and calculated, ultimately achieving natural conversation between hearing-impaired and hearing-normal people.

[1073] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1074] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts the voice input of hearing-impaired people into text data and the text data input of hearing-impaired people into voice data, but also analyzes and conveys emotional information, enabling more natural and rich communication. Here, the detailed operation of this system is explained in natural language.

[1075] Overall system configuration

[1076] The system consists of the following main components:

[1077] Server: The generation AI, chat generation AI, and emotion engine run, converting voice data and text data.

[1078] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[1079] Users: Deaf and hearing impaired people who use the system.

[1080] Voice input and emotion recognition

[1081] First, a person with normal hearing speaks into the device to input voice. "Hello. How are you today?" the person says. This voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[1082] The voice data received by the server is analyzed by an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. The emotional information recognized by the emotion engine is added as a tag to the converted text data. For example, it may be in the form of "Hello. How are you today? (Joy)."

[1083] Displaying text data and conveying emotions

[1084] The emotion-tagged text data generated by the server is sent over the network to the hearing-impaired person's device. The device decodes the received text data and displays "Hello. How are you today?" on the screen, along with visually displaying emotion information using icons and colors that indicate the emotion tag. This allows the hearing-impaired person to understand the emotions of people without hearing.

[1085] Text input and emotion communication for the hearing impaired

[1086] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This input text data is sent to the server along with the emotion information.

[1087] Text-to-speech and emotional response

[1088] The text data received by the server is passed to a TTS (Text to Speech) engine along with emotional information. The TTS engine analyzes the text data and generates voice data, "Hello. It's nice weather today and I'm feeling great," based on the emotional information. This voice data is given an intonation that corresponds to the emotional information. For example, if the emotion tag is "joy," a lively and cheerful voice is generated.

[1089] Specific examples

[1090] Here is a concrete example of a conversation between a hearing person and a deaf person:

[1091] 1. Speech input and emotion recognition for healthy subjects:

[1092] A healthy person speaks into the device: "Hello. How are you today?" (joy)

[1093] The terminal transmits the voice data to the server.

[1094] The server analyzes the voice data and recognizes the emotion as "joy."

[1095] The voice data is converted into text "Hello. How are you today?" and an emotion tag (joy) is assigned.

[1096] Text and emotional information is sent to the device of a hearing-impaired person and displayed visually.

[1097] 2. Text replies and emotional reflection for the deaf:

[1098] A deaf person types text into a device: "Hello. The weather is nice today and I'm feeling great." (Joy)

[1099] The device sends text data and emotion information to the server.

[1100] The server analyzes the text data and generates a voice message based on the emotional information: "Hello. The weather is nice today and I'm in a good mood."

[1101] The emotion of "joy" is reflected in the audio data and played back on a device belonging to a healthy person.

[1102] This system will enable more natural and rich communication between hearing-impaired and hearing-suffering people, incorporating emotional information. The introduction of an emotion engine will enable dialogue to go beyond simply exchanging information, and involve the transmission of emotions, deepening mutual understanding.

[1103] The processing flow will be explained below.

[1104] Step 1:

[1105] A user (with normal hearing) speaks to the device. The voice input device captures the user's speech and processes it as digital voice data.

[1106] Step 2:

[1107] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[1108] Step 3:

[1109] The server decodes the voice data received from the device and passes it to the emotion engine, which analyzes the voice data and recognizes the user's emotion (e.g., "happiness," "sadness," "anger," etc.).

[1110] Step 4:

[1111] The server receives the emotion information from the emotion engine and passes the voice data to the ASR (Automatic Speech Recognition) engine, which analyzes the voice data and generates corresponding text data (e.g., "Hello. How are you today?").

[1112] Step 5:

[1113] The server tags the generated text data with emotional information recognized by the emotion engine (e.g., "Hello. How are you today? (Joy)").

[1114] Step 6:

[1115] The server encodes the text data with emotion tags and sends it over the network to the hearing-impaired person's device. The text data is packetized and sent according to the appropriate communication protocol.

[1116] Step 7:

[1117] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The emotion tag (joy) is visually displayed using icons and colors.

[1118] Step 8:

[1119] A user (who is hearing impaired) uses the device's text input device to type a reply (e.g., "Hello. I'm in a good mood today.") and also types an emotion tag (joy).

[1120] Step 9:

[1121] The device receives the input text data and emotion tag, and transmits them to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[1122] Step 10:

[1123] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine based on the emotion tag. The TTS engine analyzes the text data and generates corresponding voice data (e.g., "Hello. I'm in a good mood today because the weather is nice.").

[1124] Step 11:

[1125] The server adds emotional intonation to the voice data based on the emotion tag (joy), generating voice data that reflects the emotion.

[1126] Step 12:

[1127] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[1128] Step 13:

[1129] The terminal (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. It's nice weather today and I'm feeling good" using a speaker. The voice reflects emotional intonation, and able-bodied people can understand this voice.

[1130] These are the specific processing steps of a system that combines an emotion engine. This processing enables more natural and richer dialogue that includes emotional information.

[1131] Example 2

[1132] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1133] Conventional communication systems have difficulty in achieving natural dialogue, including emotional content, between hearing-impaired and hearing-challenged people. Simply converting the speech input of hearing-impaired people into text and the text input of hearing-impaired people into speech results in the loss of emotional nuances and the resulting dialogue becoming monotonous. This drawback is particularly serious in communication where sharing emotions is important, making it difficult for people to understand each other's emotions.

[1134] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing emotional information from the received voice data, means for converting the voice data with added emotional information into text data, means for transmitting the converted text data and emotional information, means for receiving text data entered by the user together with the emotional information, means for converting the received text data and emotional information into voice data, and means for transmitting the converted voice data and emotional information. This enables more natural and rich communication, including emotion, between hearing-impaired people and those with hearing disabilities.

[1135] The "means for receiving voice data" refers to a mechanism for inputting voice data generated from an input device of a healthy person into a server.

[1136] "Means for analyzing emotion information" refers to a mechanism for identifying the user's emotion from received voice data using an emotion engine and extracting that information.

[1137] "Means for converting voice data into text data" refers to a mechanism for converting received and analyzed voice data into text information via an ASR (automatic speech recognition) engine.

[1138] "Means for transmitting text data to which emotional information has been added" refers to a mechanism for adding emotional information to converted text data and transmitting it to another terminal via a network.

[1139] The "means for receiving text data entered by a user together with emotion information" refers to a mechanism for capturing the character information entered by a user into a terminal and the selected emotion tag into a server.

[1140] "Means for converting text data and emotional information into voice data" refers to a mechanism for converting received text data and emotional information into voice data via a TTS (Text to Speech Synthesis) engine.

[1141] The "means for transmitting the converted voice data and emotional information" refers to a mechanism for transmitting the generated voice data and its emotional information to another terminal via a network.

[1142] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts voice data into text data but also analyzes and transmits emotional information, enabling more natural and rich communication.

[1143] Voice input and emotion recognition

[1144] When a user (a person with normal hearing) speaks into the device, their voice is captured using a microphone and processed as digital voice data. For example, a person with normal hearing might say, "Hello. How are you today?" This voice data is sent to a server via a network.

[1145] Audio data processing on the server

[1146] The server analyzes the received voice data using an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. Emotional information is added to the converted text data as a tag. For example, it might look like "Hello. How are you today? (Joy)". This generates data that includes text data and emotional information.

[1147] Displaying text data and conveying emotions

[1148] The device receives the emotion-tagged text data sent from the server and displays it on the screen, while visually displaying emotion information using icons and colors that indicate the emotion tag. This allows hearing-impaired people to understand the emotions of hearing-impaired people.

[1149] Text input and emotion communication for the hearing impaired

[1150] Next, the hearing-impaired person uses their own device to input a reply text. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This text data and emotional information are sent to the server via the network.

[1151] Text data processing on the server

[1152] The text data received by the server is passed to a TTS (Text to Speech) engine. The TTS engine analyzes the text data and generates voice data based on the emotional information. For example, if an emotional tag (joy) is attached, a lively and cheerful voice saying "Hello. The weather is nice today and I'm in a good mood" is generated.

[1153] Playing audio data

[1154] The generated voice data is sent to the device of the hearing-impaired person and played back, allowing the hearing-impaired person to understand the emotions of the hearing-impaired person.

[1155] Specific examples

[1156] Here are some examples of specific prompts:

[1157] 1. "Turn the following speech data into text and analyze the user's sentiment: Hello. How are you today?"

[1158] 2. "Generate speech from emotion-tagged text data: Hello. The weather is nice today and I'm feeling great. (Joy)"

[1159] Hardware and software used

[1160] Devices: Smartphones, tablets, PCs, etc.

[1161] Server: Generation AI, chat generation AI, emotion engine, ASR engine, TTS engine

[1162] Emotion Recognition Software: Emotion Engine

[1163] Automatic Speech Recognition Software: ASR Engine

[1164] Text to Speech Software: TTS Engine

[1165] By using this system, communication between hearing-impaired and hearing-challenged people can take place in a way that includes emotional information, enabling a more natural and deeper mutual understanding.

[1166] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1167] Step 1:

[1168] Device:

[1169] A user (healthy person) speaks into the device. Example: "Hello. How are you today?"

[1170] Input: Normal human speech

[1171] How it works: The device's microphone captures audio and converts it into audio data.

[1172] Output: Digital audio data

[1173] Step 2:

[1174] Device:

[1175] Input: Captured digital audio data

[1176] What it does: The device sends this digital audio data to the server.

[1177] Output: Audio data sent to the server

[1178] Step 3:

[1179] server:

[1180] Input: Transmitted audio data

[1181] How it works: The server analyzes this voice data through an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger).

[1182] Output: Audio data with emotional information added

[1183] Step 4:

[1184] server:

[1185] Input: Audio data with emotional information

[1186] How it works: This data is converted into text using an ASR (automatic speech recognition) engine.

[1187] Output: Emotion-tagged text data (e.g., "Hello. How are you today? (Joy)")

[1188] Step 5:

[1189] server:

[1190] Input: Emotion-tagged text data

[1191] How it works: The server sends the generated emotion-tagged text data to the hearing-impaired person's device via the network.

[1192] Output: Text data and emotional information sent to the hearing-impaired person's device

[1193] Step 6:

[1194] Terminal (hearing impaired):

[1195] Input: Emotion-tagged text data

[1196] How it works: The device displays the received text data on the screen and visually represents the emotional information using icons and colors.

[1197] Output: Text and emotion information displayed on the screen

[1198] Step 7:

[1199] User (hearing impaired):

[1200] Input: Terminal input field

[1201] Action: A deaf person types a reply. Example: "Hello. The weather is nice today and I'm feeling great."

[1202] Output: Input text data

[1203] Step 8:

[1204] Terminal (hearing impaired):

[1205] Input: Text data entered by a hearing-impaired person

[1206] What it does: Sends the text and the selected emotion tag to the server.

[1207] Output: Text data and emotion information sent to the server

[1208] Step 9:

[1209] server:

[1210] Input: Received text data and emotion information

[1211] How it works: The server passes this to a TTS (Text to Speech) engine, which converts the received text data into voice data. At that time, it adds an appropriate tone to the voice based on the emotion tag. For example, for the (joy) tag, a lively and cheerful tone

[1212] Output: Audio data reflecting emotional information

[1213] Step 10:

[1214] server:

[1215] Input: Voice data reflecting emotional information

[1216] Operation: The server transmits the generated voice data to the device of the healthy person via the network.

[1217] Output: Audio data sent to the healthy subject's device

[1218] Step 11:

[1219] Device (healthy person):

[1220] Input: Transmitted audio data

[1221] Action: Plays back the audio data and has a normal subject listen to it. Example: "Hello. The weather is nice today and I'm feeling great."

[1222] Output: Played audio data

[1223] Through these steps, natural communication, including emotional content, is realized between hearing-impaired and hearing-challenged people.

[1224] (Application example 2)

[1225] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1226] It is difficult to achieve smooth and natural communication between hearing-impaired and hearing-savvy people. Conventional systems only convert voice data to text data, and lack emotional exchange in communication. This leads to insufficient communication and a decline in the quality of communication. Especially in brick-and-mortar stores, there is a strong demand for communication that includes emotional information, as this has a significant impact on customer experience and service quality.

[1227] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, emotion analysis means for analyzing the user's emotion and generating emotion information, means for assigning the generated emotion information to the text data, means for visualizing the assigned emotion information, and means for reflecting the assigned emotion information in the voice data. This enables natural and rich communication including emotion information between hearing-impaired and hearing-normal people.

[1228] "Means for receiving audio data" refers to a device or method that receives an external audio signal with low loss and converts it into a format suitable for internal processing.

[1229] A "means for converting received voice data into text data" is a device or method that uses voice recognition technology to convert received voice signals into a corresponding text format.

[1230] The "means for transmitting converted text data" refers to a device or method for transmitting the converted text data to another device or system.

[1231] The "means for receiving text data entered by a user" refers to a device or method that receives text data entered by a user and converts it into a format suitable for internal processing.

[1232] A "means for converting received text data into speech data" is a device or method that uses text-to-speech technology to convert text data into a corresponding speech format.

[1233] The "means for transmitting converted audio data" refers to a device or method for transmitting the converted audio data to another device or system.

[1234] "Emotion analysis means" refers to a device or method that reads the user's emotions from input voice or text data and generates emotion information.

[1235] The "means for adding generated emotion information to text data" refers to a device or method for adding emotion information obtained by emotion analysis to text and reinforcing the text data.

[1236] The "means for visualizing assigned emotional information" is a device or method for visualizing analyzed emotional information and presenting it to the user.

[1237] The "means for reflecting the added emotional information in the voice data" refers to a device or method for incorporating the generated emotional information into the voice synthesis process and expressing it as voice.

[1238] Overall system configuration

[1239] This system aims to realize seamless communication between hearing-impaired and hearing-suffering people. The system is broadly composed of the following elements:

[1240] Server: The emotion analysis, speech recognition, and text-to-speech synthesis engines run on a high-performance cloud server.

[1241] Terminal: A device capable of voice input / output and text display, such as a smartphone or smart glasses.

[1242] Users: Deaf and hearing impaired people who use the system.

[1243] Hardware / Software used

[1244] Server: A cloud server such as AWS EC2 or Google Cloud.

[1245] Emotion engine: Microsoft Azure Emotion API.

[1246] Speech recognition engine (ASR): Google Cloud Speech-to-Text.

[1247] Text-to-speech engine (TTS): Amazon Polly.

[1248] Program processing explanation

[1249] Voice input and emotion recognition

[1250] 1. Terminal: Voice input is performed by a healthy person speaking into the terminal.

[1251] 2. Server: The audio data captured on the device is sent to the server.

[1252] 3. Server (emotion analysis means): The received voice data is analyzed by the emotion engine, and emotional information such as joy, sadness, and anger is generated.

[1253] 4. Server (voice recognition means): Voice data is converted into text data by the ASR engine.

[1254] 5. Server (means for adding emotional information): Emotional information is added to text data as tags. For example, "Hello. How are you today? (Joy)".

[1255] Displaying text data and conveying emotions

[1256] 6. Server: The generated emotion-tagged text data is sent to the hearing-impaired person's device.

[1257] 7. Terminal (visualization means): The text data is decoded and displayed on the screen, and emotion tags are visually displayed using icons and colors.

[1258] Text input and emotion communication for the hearing impaired

[1259] 8. Device: A hearing impaired person uses a device to input text, for example, "Hello. The weather is nice today and I feel good. (Joy)."

[1260] 9. Server: The input text data is sent to the server along with the emotion information.

[1261] Text-to-speech and emotional response

[1262] 10. Server (text-to-speech synthesis means): The received text data is passed to the TTS engine, which analyzes the text data and generates voice data.

[1263] 11. Server (Means for Reflecting Emotional Information): The server adds intonation to the voice data based on the emotional information. For example, if the voice data is tagged with "joy," it generates a voice with a lively and cheerful tone.

[1264] 12. Terminal: The generated voice data is played on the terminal of the able-bodied person.

[1265] Examples of concrete examples and prompts

[1266] Here is a concrete example of the dialogue:

[1267] 1. Healthy person: "Hello. How are you today?" (joy).

[1268] 2. Hearing impaired person: "Hello. The weather is nice today and I feel great." (joy).

[1269] Example prompt for a generative AI model:

[1270] "The store clerk greeted: Hello, is there anything you're looking for?"

[1271] "Customer enters through smart glasses: I'm not looking for anything in particular today."

[1272] "Convert these messages into text and audio with appropriate emotional information."

[1273] This system enables natural and rich communication, including emotional information, between hearing-impaired and hearing-savvy people. Specifically, it is expected to improve the customer experience and service quality in brick-and-mortar stores.

[1274] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1275] Step 1:

[1276] Device: Voice input is performed by a non-disabled person speaking into the device. The device's built-in microphone captures the voice signal and stores it as digitally formatted voice data. This voice data is sent to a server for further processing. The input here is the voice spoken by a non-disabled person, and the output is digitally formatted voice data.

[1277] Step 2:

[1278] Server: Receives the voice data sent from the device. The server checks the format of the voice data and converts it to the appropriate format if necessary. At this stage, data pre-processing is performed, such as noise filtering. The input is voice data, and the output is pre-processed voice data.

[1279] Step 3:

[1280] Server (emotion analysis means): Analyzes the received voice data using an emotion engine to generate emotional information for healthy individuals. In the analysis step, emotions are estimated based on the pitch, tone, and speed of the voice. The input is processed voice data, and the output is metadata containing emotional information.

[1281] Step 4:

[1282] Server (voice recognition means): After emotion analysis, the voice data is converted into text data by the ASR engine. The voice recognition engine analyzes the voice waveform and converts it into the corresponding text format. The input here is the voice data that has been emotion analyzed, and the output is the converted text data.

[1283] Step 5:

[1284] Server (emotion information adding means): Adds the generated emotion information to the converted text data as a tag. Specifically, emotion information is added to the end of a sentence in the text. For example, "Hello. How are you today? (Joy)". The input is text data and emotion metadata, and the output is emotion-tagged text data.

[1285] Step 6:

[1286] Server: Sends emotion-tagged text data to the device of the hearing-impaired person. Data transfer takes place here, and stable communication is ensured using network protocols (e.g., HTTP, WebSocket). The input is emotion-tagged text data, and the output is the text data received by the device of the hearing-impaired person.

[1287] Step 7:

[1288] Terminal (Visualization Means): The terminal of the hearing-impaired person decodes the received text data and displays it on the screen. At the same time, visual feedback (e.g., icons or colors) based on the emotion tag is also displayed. The input is emotion-tagged text data, and the output is the displayed text and emotion information.

[1289] Step 8:

[1290] User: A hearing-impaired person uses the device to input a text message. For example, "Hello. The weather is nice today and I feel great. (Joy)." The device receives the user's input and stores it as text data. The input is the text entered by the user, and the output is text data.

[1291] Step 9:

[1292] Terminal: Sends input text data to the server. The terminal's communication module sends the text data to the server, and the data is encrypted if necessary. The input is text data, and the output is the text data sent to the server.

[1293] Step 10:

[1294] Server (Text-to-Speech Synthesis Means): The received text data is converted into voice data by the TTS engine. The TTS engine analyzes the text and generates natural-sounding voice. The input is text data, and the output is the generated voice data.

[1295] Step 11:

[1296] Server (emotional information reflection means): Reflects emotional information in the generated voice data. Specifically, it adjusts the tone and pitch of the voice based on the emotion tag. The input is voice data and emotional information, and the output is voice data that reflects the emotional information.

[1297] Step 12:

[1298] Terminal: The audio data reflecting emotional information is played on the terminal of the hearing-impaired person. Here, the hearing-impaired person can receive a message from the hearing-impaired person by voice. The input is the audio data reflecting emotional information, and the output is the audio played from the terminal.

[1299] In this way, the system can realize natural communication including emotional information between hearing-impaired and hearing-normal people.

[1300] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1301] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1302] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1303] [Fourth embodiment]

[1304] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1305] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1306] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1307] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1308] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1309] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1310] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1311] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1312] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1313] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1314] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1315] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1316] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1317] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of hearing-impaired people into text data, and converts the text data input by hearing-impaired people into voice data, enabling natural dialogue in real time. Here, the detailed operation of this system is explained in natural language.

[1318] Overall system configuration

[1319] The system consists of the following main components:

[1320] Server: The generation AI and chat generation AI run, converting voice data and text data.

[1321] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[1322] Users: Deaf and hearing impaired people who use the system.

[1323] Processing voice input

[1324] First, a person with normal hearing speaks into the device to input voice. When a person with normal hearing speaks, "Hello. How are you today?", the voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[1325] Speech-to-text conversion

[1326] The voice data received by the server is processed by the ASR (Automatic Speech Recognition) engine, a component of the generative AI. This engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?" This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[1327] Text entry for the hearing impaired

[1328] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling great." This input text data is sent to the server by the device.

[1329] Text to speech conversion

[1330] The text data received by the server is processed by a TTS (Text to Speech) engine, a component of the chat generation AI. This engine analyzes the text data and generates the corresponding voice data, "Hello. The weather is nice today and I'm in a good mood." This generated voice data is sent to the device of the able-bodied person and played back through the speaker.

[1331] Specific examples

[1332] Here is a concrete example of a conversation between a hearing person and a deaf person:

[1333] 1. Voice input from a healthy subject:

[1334] A healthy person speaks into the device: "Hello. How are you today?"

[1335] The terminal transmits the voice data to the server.

[1336] The server converts the voice data into text "Hello, how are you today?"

[1337] The text appears on the device of the hearing impaired person.

[1338] 2. Text Response for the Deaf:

[1339] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[1340] The terminal transmits the text data to the server.

[1341] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[1342] The device of the able-bodied person plays the audio.

[1343] This system will enable natural and seamless communication between hearing-impaired and hearing-suffering people. This invention will significantly improve the communication environment for hearing-impaired people, enabling both parties to communicate efficiently.

[1344] The processing flow will be explained below.

[1345] Step 1:

[1346] A user (with normal hearing) speaks to the terminal. The voice input device captures the user's speech, "Hello. How are you today?" and processes it as digital voice data.

[1347] Step 2:

[1348] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[1349] Step 3:

[1350] The server decodes the voice data received from the device and passes it to the ASR (Automatic Speech Recognition) engine for processing. The ASR engine analyzes the voice data and generates the corresponding text data: "Hello. How are you today?"

[1351] Step 4:

[1352] The server encodes the generated text data and transmits it over the network to the hearing-impaired user's device. The text data is packetized and transmitted according to the appropriate communication protocol.

[1353] Step 5:

[1354] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The hearing impaired person reads and understands this text.

[1355] Step 6:

[1356] A user (who is hearing impaired) uses the text input device of the device to input a reply, for example, "Hello. The weather is nice today and I'm in a good mood."

[1357] Step 7:

[1358] The terminal receives the input text data and transmits it to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[1359] Step 8:

[1360] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine for processing. The TTS engine analyzes the text data and generates the corresponding voice data: "Hello. The weather is nice today and I'm in a good mood."

[1361] Step 9:

[1362] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[1363] Step 10:

[1364] The device (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. The weather is nice today and I'm feeling great." using the device's speaker. A able-bodied person listens to and understands this voice.

[1365] These are the specific processing steps of the system, which allows for seamless communication between hearing-impaired and hearing-savvy people.

[1366] Example 1

[1367] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1368] In communication between hearing-impaired and hearing-savvy people, there is a need for an efficient and accurate system that can convert speech to text and vice versa in real time to achieve seamless dialogue. Conventional systems have insufficient conversion accuracy and speed, often causing stress and discomfort during actual dialogue. In addition, the need to use multiple different devices and applications makes operation cumbersome, and there have been few easy-to-use systems.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1370] In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, means for transmitting the converted text data, means for receiving text data entered by a user, means for converting the received text data into voice data, means for transmitting the converted voice data, means including an automatic speech recognition engine using a generative AI model, and means including a text-to-speech synthesis engine using a generative AI model, thereby enabling real-time and seamless dialogue between hearing-impaired people and hearing-normal people.

[1371] "Audio data" is data that represents an audio signal in digital form.

[1372] "Text data" is data that represents character information in digital form.

[1373] A "generative AI model" is an artificial intelligence model used to automatically generate and transform data.

[1374] An "automatic speech recognition engine" is a software component that analyzes speech data and converts it into corresponding text data.

[1375] A "text-to-speech engine" is a software component that analyzes text data and converts it into corresponding speech data.

[1376] "Network connectivity" is the infrastructure for communicating data between devices.

[1377] "User" refers to an individual who uses the system, and includes both hearing-impaired and hearing-nondisabled individuals.

[1378] This invention is a system for realizing seamless communication between hearing-impaired and hearing-normal people. This system converts the voice input of the hearing-impaired person into text data, and converts the text data input by the hearing-impaired person into voice data, enabling natural dialogue in real time.

[1379] Overall system configuration

[1380] The system consists of the following main components:

[1381] Server: The generative AI model and its components, the automatic speech recognition engine (ASR) and the text-to-speech synthesis engine (TTS), run here, converting voice data and text data.

[1382] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[1383] Users: Deaf and hearing impaired people who use the system.

[1384] Hardware and software used

[1385] Hardware: Various devices such as a microphone, speaker, display, keyboard, and network interface are required. Examples include smartphones, tablets, and PCs.

[1386] Software: Includes software components such as generative AI models, ASR engines (e.g., Google Cloud Speech-to-Text, Microsoft Azure Speech Services), and TTS engines (e.g., Google Cloud Text-to-Speech, AWS Polly).

[1387] Data processing and data calculation

[1388] The server receives the voice data sent from the device, analyzes it with an ASR engine, and converts it into text data. This text data is then sent back to the device and displayed on the screen. Conversely, the text data sent from the device is converted into voice data by the server with a TTS engine, and played back on the device. This series of processes enables seamless communication between hearing-impaired and hearing-normal people.

[1389] Specific examples

[1390] Here is a concrete example of a conversation between a hearing person and a deaf person:

[1391] 1. Voice input from a healthy subject:

[1392] A healthy person speaks into the device: "Hello. How are you today?"

[1393] The terminal transmits the voice data to the server.

[1394] The server converts the voice data into text "Hello, how are you today?"

[1395] The text appears on the device of the hearing impaired person.

[1396] 2. Text Response for the Deaf:

[1397] A deaf person types a text into their device: "Hello. The weather is nice today and I'm feeling great."

[1398] The terminal transmits the text data to the server.

[1399] The server converts the text data into speech, "Hello. The weather is nice today and I'm in a good mood."

[1400] The device of the able-bodied person plays the audio.

[1401] Prompt Sentence Examples

[1402] "A hearing-impaired person inputs speech, which is converted into text and displayed to a hearing-impaired person. Please explain in detail the system that inputs this text, converts it back into speech, and plays it back to the hearing-impaired person."

[1403] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people. By utilizing generative AI models, highly accurate and real-time dialogue is possible, significantly removing communication barriers.

[1404] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1405] (Details of processing steps)

[1406] Step 1:

[1407] Voice input and capture for able-bodied people

[1408] Input: A healthy person speaks into the terminal.

[1409] Specific action: A healthy person says, "Hello. How are you today?"

[1410] Data processing and output: The microphone on the device captures the audio and stores it in a temporary buffer as digital audio data. This audio data is then sent to the server via the network.

[1411] Step 2:

[1412] Speech-to-text conversion

[1413] Input: Digital audio data sent from the device.

[1414] Specific operation: The server receives the audio data.

[1415] Data processing and output: An automatic speech recognition engine (ASR), part of the generative AI model on the server, analyzes the voice data and converts it into corresponding text data: "Hello. How are you today?" This text data is stored in a temporary database and sent to the hearing-impaired person's device.

[1416] Step 3:

[1417] Displaying Text

[1418] Input: Text data sent from the server.

[1419] Specific operation: The hearing-impaired person's device receives text data.

[1420] Data processing and output: Displaying text on the terminal display. For example, updating the text "Hello, how are you today?" in the display area.

[1421] Step 4:

[1422] Text entry for the hearing impaired

[1423] Input: A hearing-impaired person types text into a terminal.

[1424] Specific action: A hearing-impaired person types, "Hello. The weather is nice today and I'm feeling great."

[1425] Data processing and output: The input text data is temporarily stored on the terminal and sent to the server via the network.

[1426] Step 5:

[1427] Text to speech conversion

[1428] Input: Text data sent from the terminal.

[1429] Specific operation: The server receives the text data.

[1430] Data processing and output: A text-to-speech engine (TTS), part of the generative AI model on the server, analyzes the text data and converts it into corresponding voice data: "Hello. The weather is nice today and I'm feeling great." This voice data is stored in a temporary database and sent to the able-bodied user's device.

[1431] Step 6:

[1432] Playing audio

[1433] Input: Audio data sent from the server.

[1434] Specific operation: The device of the healthy person receives the voice data.

[1435] Data processing and output: The received voice data is played back through the device's speaker. For example, a healthy person hears a voice saying, "Hello. The weather is nice today and I'm feeling good."

[1436] By inputting, processing, and outputting data at each step, seamless real-time communication between hearing-impaired and hearing-speaking people is realized. The system uses speech recognition and speech synthesis technology that utilizes generative AI models to provide high accuracy and smooth dialogue.

[1437] (Application example 1)

[1438] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1439] In modern society, there are limited ways for hearing-impaired and hearing-normal people to communicate effectively. In particular, in brick-and-mortar stores, hearing-impaired customers often have difficulty communicating with store staff, which can affect the quality of service. To solve this problem, a system is needed that allows both hearing-impaired and hearing-normal people to communicate naturally and efficiently.

[1440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1441] In this invention, the server includes a means for receiving voice data, a means for converting the received voice data into text data, and a means for transmitting the converted text data, thereby enabling two-way communication of text data and voice data between customers and store clerks in a physical store.

[1442] The "means for receiving audio data" is a device or function that acquires audio input from the outside as digital data.

[1443] The "means for converting received voice data into text data" refers to a device or function that analyzes voice data and converts it into corresponding text data.

[1444] The "means for transmitting converted text data" refers to a device or function that transmits the generated text data to another device or system.

[1445] The "means for receiving text data entered by a user" refers to a device or function that acquires text data entered by a user.

[1446] The "means for converting received text data into voice data" refers to a device or function that analyzes text data and converts it into corresponding voice data.

[1447] The "means for transmitting converted audio data" is a device or function that transmits the generated audio data to another device or system.

[1448] "Means for enabling two-way communication of text data and voice data between customers and store clerks in a physical store" refers to a device or function that allows customers and store clerks in a physical store to exchange information in text and voice format two-way.

[1449] This invention is a system that enables seamless communication between hearing-impaired people and hearing-normal people (store clerks) in brick-and-mortar stores. This system converts voice data into text data and text data into voice data, enabling natural dialogue in real time.

[1450] Overall system configuration

[1451] The system consists of the following main components:

[1452] Server: The server runs a speech recognition engine, a text-to-speech synthesis engine, and a data transmission / reception module, and converts voice data into text data.

[1453] Terminal: A device used by users (hearing impaired and hearing-able) that is capable of voice input / output, and text input / display, such as a smartphone.

[1454] Users: Deaf and hearing-impaired people (store clerks) who use the system.

[1455] Processing voice input

[1456] First, a person with normal hearing speaks into the device to input voice. For example, if a store clerk says, "Are you looking for something?", this voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[1457] Speech-to-text conversion

[1458] The voice data received by the server is analyzed by a voice recognition engine, and the corresponding text data, "Are you looking for something?", is generated. This generated text data is sent to the hearing-impaired person's device and displayed on the screen.

[1459] Text entry for the hearing impaired

[1460] Next, the hearing-impaired person uses their own device to input a reply, for example, "Yes, I'm looking for shoes." This input text data is sent to the server by the device.

[1461] Text to speech conversion

[1462] The text data received by the server is analyzed by a text-to-speech synthesis engine, which generates the corresponding voice data, "Hello, I'm looking for shoes." This voice data is sent to the device of the able-bodied person and played back through a speaker.

[1463] Specific examples

[1464] For example, here is a concrete example of a conversation between a hearing person and a hearing-impaired person:

[1465] 1. Voice input from a healthy subject:

[1466] Store clerk: "Are you looking for something?"

[1467] The terminal transmits the voice data to the server.

[1468] The server converts the voice data into the text "Are you looking for something?"

[1469] The text appears on the device of the hearing impaired person.

[1470] 2. Text Response for the Deaf:

[1471] Deaf person: "Yes, I'm looking for my shoes."

[1472] The terminal transmits the text data to the server.

[1473] The server converts the text data into speech, "Yes, I'm looking for shoes."

[1474] The clerk's terminal plays the audio.

[1475] This system enables natural and seamless communication between hearing-impaired and hearing-savvy people in brick-and-mortar stores. This invention significantly improves the environment in which hearing-impaired people receive services in brick-and-mortar stores, enabling both parties to communicate efficiently.

[1476] Prompt Sentence Examples

[1477] "Hello. How are you today? Is there anything I can help you with?"

[1478] The above configuration provides an environment in which hearing-impaired and hearing-normal people can communicate smoothly in physical stores.

[1479] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1480] Step 1:

[1481] Voice input is performed when a person with hearing loss speaks into the device. The device captures this voice input as digital voice data. The input is what the person with hearing loss said (voice), and the output is digital voice data. This digital voice data is temporarily stored in the device.

[1482] Step 2:

[1483] The terminal transmits the captured audio data to the server via the network. The input is the captured digital audio data, and the output is the audio data transmitted to the server. This is where the data transmission module operates.

[1484] Step 3:

[1485] The speech recognition engine analyzes the voice data received by the server and converts it into corresponding text data. The input is the received digital voice data and the output is the corresponding text data. The speech recognition engine uses a generative AI model to analyze the voice and generate accurate text.

[1486] Step 4:

[1487] The server sends the generated text data to the terminal of the hearing impaired person. The input is the converted text data, and the output is the text data sent to the terminal of the hearing impaired person. Here, the data sending module is operated again.

[1488] Step 5:

[1489] The hearing impaired person's device displays the received text data on the screen. The input is the received text data, and the output is the text displayed on the display screen. The display module of the device operates to provide the text visually.

[1490] Step 6:

[1491] A hearing-impaired person types a text reply into their device. The input is text data entered by the hearing-impaired person, and the output is text data stored on the device. This text is entered using the device's input device (keyboard or touch screen).

[1492] Step 7:

[1493] The terminal sends the input text data to the server. The input is the text data entered by the hearing impaired person, and the output is the text data sent to the server. The terminal transfers the data via the network.

[1494] Step 8:

[1495] The text data received by the server is analyzed by the text-to-speech engine and converted into corresponding voice data. The input is the received text data and the output is the generated voice data. The text-to-speech engine uses a generative AI model to generate natural-sounding voices.

[1496] Step 9:

[1497] The server sends the generated voice data to the device of the able-bodied person. The input is the generated voice data, and the output is the voice data sent to the device of the able-bodied person. Here too, the data transmission module operates.

[1498] Step 10:

[1499] The device of the able-bodied person plays the received voice data. The input is the received voice data, and the output is the voice to be played. The speaker module of the device operates and provides the voice to the able-bodied person.

[1500] These are the specific processing steps in this system. At each step, data is processed and calculated, ultimately achieving natural conversation between hearing-impaired and hearing-normal people.

[1501] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1502] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts the voice input of hearing-impaired people into text data and the text data input of hearing-impaired people into voice data, but also analyzes and conveys emotional information, enabling more natural and rich communication. Here, the detailed operation of this system is explained in natural language.

[1503] Overall system configuration

[1504] The system consists of the following main components:

[1505] Server: The generation AI, chat generation AI, and emotion engine run, converting voice data and text data.

[1506] Terminal: A device used by both hearing-impaired and hearing-normal people, capable of voice input / output and text input / display.

[1507] Users: Deaf and hearing impaired people who use the system.

[1508] Voice input and emotion recognition

[1509] First, a person with normal hearing speaks into the device to input voice. "Hello. How are you today?" the person says. This voice is captured by the device and processed as digital voice data. The device then transmits this voice data to a server via the network.

[1510] The voice data received by the server is analyzed by an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. The emotional information recognized by the emotion engine is added as a tag to the converted text data. For example, it may be in the form of "Hello. How are you today? (Joy)."

[1511] Displaying text data and conveying emotions

[1512] The emotion-tagged text data generated by the server is sent over the network to the hearing-impaired person's device. The device decodes the received text data and displays "Hello. How are you today?" on the screen, along with visually displaying emotion information using icons and colors that indicate the emotion tag. This allows the hearing-impaired person to understand the emotions of people without hearing.

[1513] Text input and emotion communication for the hearing impaired

[1514] Next, the hearing-impaired person uses their own device to input a reply. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This input text data is sent to the server along with the emotion information.

[1515] Text-to-speech and emotional response

[1516] The text data received by the server is passed to a TTS (Text to Speech) engine along with emotional information. The TTS engine analyzes the text data and generates voice data, "Hello. It's nice weather today and I'm feeling great," based on the emotional information. This voice data is given an intonation that corresponds to the emotional information. For example, if the emotion tag is "joy," a lively and cheerful voice is generated.

[1517] Specific examples

[1518] Here is a concrete example of a conversation between a hearing person and a deaf person:

[1519] 1. Speech input and emotion recognition for healthy subjects:

[1520] A healthy person speaks into the device: "Hello. How are you today?" (joy)

[1521] The terminal transmits the voice data to the server.

[1522] The server analyzes the voice data and recognizes the emotion as "joy."

[1523] The voice data is converted into text "Hello. How are you today?" and an emotion tag (joy) is assigned.

[1524] Text and emotional information is sent to the device of a hearing-impaired person and displayed visually.

[1525] 2. Text replies and emotional reflection for the deaf:

[1526] A deaf person types text into a device: "Hello. The weather is nice today and I'm feeling great." (Joy)

[1527] The device sends text data and emotion information to the server.

[1528] The server analyzes the text data and generates a voice message based on the emotional information: "Hello. The weather is nice today and I'm in a good mood."

[1529] The emotion of "joy" is reflected in the audio data and played back on a device belonging to a healthy person.

[1530] This system will enable more natural and rich communication between hearing-impaired and hearing-suffering people, incorporating emotional information. The introduction of an emotion engine will enable dialogue to go beyond simply exchanging information, and involve the transmission of emotions, deepening mutual understanding.

[1531] The processing flow will be explained below.

[1532] Step 1:

[1533] A user (with normal hearing) speaks to the device. The voice input device captures the user's speech and processes it as digital voice data.

[1534] Step 2:

[1535] The terminal receives the voice data and transmits it to the server via the network. The voice data is packetized and sent according to the appropriate communication protocol.

[1536] Step 3:

[1537] The server decodes the voice data received from the device and passes it to the emotion engine, which analyzes the voice data and recognizes the user's emotion (e.g., "happiness," "sadness," "anger," etc.).

[1538] Step 4:

[1539] The server receives the emotion information from the emotion engine and passes the voice data to the ASR (Automatic Speech Recognition) engine, which analyzes the voice data and generates corresponding text data (e.g., "Hello. How are you today?").

[1540] Step 5:

[1541] The server tags the generated text data with emotional information recognized by the emotion engine (e.g., "Hello. How are you today? (Joy)").

[1542] Step 6:

[1543] The server encodes the text data with emotion tags and sends it over the network to the hearing-impaired person's device. The text data is packetized and sent according to the appropriate communication protocol.

[1544] Step 7:

[1545] The terminal (hearing impaired device) decodes the text data received from the server and displays the text "Hello. How are you today?" on the screen. The emotion tag (joy) is visually displayed using icons and colors.

[1546] Step 8:

[1547] A user (who is hearing impaired) uses the device's text input device to type a reply (e.g., "Hello. I'm in a good mood today.") and also types an emotion tag (joy).

[1548] Step 9:

[1549] The device receives the input text data and emotion tag, and transmits them to the server via the network. The text data is packetized and sent according to the appropriate communication protocol.

[1550] Step 10:

[1551] The server decodes the text data received from the device and passes it to a TTS (Text to Speech) engine based on the emotion tag. The TTS engine analyzes the text data and generates corresponding voice data (e.g., "Hello. I'm in a good mood today because the weather is nice.").

[1552] Step 11:

[1553] The server adds emotional intonation to the voice data based on the emotion tag (joy), generating voice data that reflects the emotion.

[1554] Step 12:

[1555] The server encodes the generated voice data and transmits it to the healthy user's terminal via the network. The voice data is packetized and transmitted according to the appropriate communication protocol.

[1556] Step 13:

[1557] The terminal (device for able-bodied people) decodes the voice data received from the server and plays the voice "Hello. It's nice weather today and I'm feeling good" using a speaker. The voice reflects emotional intonation, and able-bodied people can understand this voice.

[1558] These are the specific processing steps of a system that combines an emotion engine. This processing enables more natural and richer dialogue that includes emotional information.

[1559] Example 2

[1560] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] Conventional communication systems have difficulty in achieving natural dialogue, including emotional content, between hearing-impaired and hearing-challenged people. Simply converting the speech input of hearing-impaired people into text and the text input of hearing-impaired people into speech results in the loss of emotional nuances and the resulting dialogue becoming monotonous. This drawback is particularly serious in communication where sharing emotions is important, making it difficult for people to understand each other's emotions.

[1562] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for analyzing emotional information from the received voice data, means for converting the voice data with added emotional information into text data, means for transmitting the converted text data and emotional information, means for receiving text data entered by the user together with the emotional information, means for converting the received text data and emotional information into voice data, and means for transmitting the converted voice data and emotional information. This enables more natural and rich communication, including emotion, between hearing-impaired people and those with hearing disabilities.

[1563] The "means for receiving voice data" refers to a mechanism for inputting voice data generated from an input device of a healthy person into a server.

[1564] "Means for analyzing emotion information" refers to a mechanism for identifying the user's emotion from received voice data using an emotion engine and extracting that information.

[1565] "Means for converting voice data into text data" refers to a mechanism for converting received and analyzed voice data into text information via an ASR (automatic speech recognition) engine.

[1566] "Means for transmitting text data to which emotional information has been added" refers to a mechanism for adding emotional information to converted text data and transmitting it to another terminal via a network.

[1567] The "means for receiving text data entered by a user together with emotion information" refers to a mechanism for capturing the character information entered by a user into a terminal and the selected emotion tag into a server.

[1568] "Means for converting text data and emotional information into voice data" refers to a mechanism for converting received text data and emotional information into voice data via a TTS (Text to Speech Synthesis) engine.

[1569] The "means for transmitting the converted voice data and emotional information" refers to a mechanism for transmitting the generated voice data and its emotional information to another terminal via a network.

[1570] This invention combines a system for achieving seamless communication between hearing-impaired and hearing-normal people with an emotion engine that recognizes the user's emotions. This system not only converts voice data into text data but also analyzes and transmits emotional information, enabling more natural and rich communication.

[1571] Voice input and emotion recognition

[1572] When a user (a person with normal hearing) speaks into the device, their voice is captured using a microphone and processed as digital voice data. For example, a person with normal hearing might say, "Hello. How are you today?" This voice data is sent to a server via a network.

[1573] Audio data processing on the server

[1574] The server analyzes the received voice data using an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger). The voice data is then converted into text data by an ASR (automatic speech recognition) engine. Emotional information is added to the converted text data as a tag. For example, it might look like "Hello. How are you today? (Joy)". This generates data that includes text data and emotional information.

[1575] Displaying text data and conveying emotions

[1576] The device receives the emotion-tagged text data sent from the server and displays it on the screen, while visually displaying emotion information using icons and colors that indicate the emotion tag. This allows hearing-impaired people to understand the emotions of hearing-impaired people.

[1577] Text input and emotion communication for the hearing impaired

[1578] Next, the hearing-impaired person uses their own device to input a reply text. For example, they input the text "Hello. It's nice weather today and I'm feeling good. (Joy)." This text data and emotional information are sent to the server via the network.

[1579] Text data processing on the server

[1580] The text data received by the server is passed to a TTS (Text to Speech) engine. The TTS engine analyzes the text data and generates voice data based on the emotional information. For example, if an emotional tag (joy) is attached, a lively and cheerful voice saying "Hello. The weather is nice today and I'm in a good mood" is generated.

[1581] Playing audio data

[1582] The generated voice data is sent to the device of the hearing-impaired person and played back, allowing the hearing-impaired person to understand the emotions of the hearing-impaired person.

[1583] Specific examples

[1584] Here are some examples of specific prompts:

[1585] 1. "Turn the following speech data into text and analyze the user's sentiment: Hello. How are you today?"

[1586] 2. "Generate speech from emotion-tagged text data: Hello. The weather is nice today and I'm feeling great. (Joy)"

[1587] Hardware and software used

[1588] Devices: Smartphones, tablets, PCs, etc.

[1589] Server: Generation AI, chat generation AI, emotion engine, ASR engine, TTS engine

[1590] Emotion Recognition Software: Emotion Engine

[1591] Automatic Speech Recognition Software: ASR Engine

[1592] Text to Speech Software: TTS Engine

[1593] By using this system, communication between hearing-impaired and hearing-challenged people can take place in a way that includes emotional information, enabling a more natural and deeper mutual understanding.

[1594] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1595] Step 1:

[1596] Device:

[1597] A user (healthy person) speaks into the device. Example: "Hello. How are you today?"

[1598] Input: Normal human speech

[1599] How it works: The device's microphone captures audio and converts it into audio data.

[1600] Output: Digital audio data

[1601] Step 2:

[1602] Device:

[1603] Input: Captured digital audio data

[1604] What it does: The device sends this digital audio data to the server.

[1605] Output: Audio data sent to the server

[1606] Step 3:

[1607] server:

[1608] Input: Transmitted audio data

[1609] How it works: The server analyzes this voice data through an emotion engine to recognize the user's emotional information (e.g., joy, sadness, anger).

[1610] Output: Audio data with emotional information added

[1611] Step 4:

[1612] server:

[1613] Input: Audio data with emotional information

[1614] How it works: This data is converted into text using an ASR (automatic speech recognition) engine.

[1615] Output: Emotion-tagged text data (e.g., "Hello. How are you today? (Joy)")

[1616] Step 5:

[1617] server:

[1618] Input: Emotion-tagged text data

[1619] How it works: The server sends the generated emotion-tagged text data to the hearing-impaired person's device via the network.

[1620] Output: Text data and emotional information sent to the hearing-impaired person's device

[1621] Step 6:

[1622] Terminal (hearing impaired):

[1623] Input: Emotion-tagged text data

[1624] How it works: The device displays the received text data on the screen and visually represents the emotional information using icons and colors.

[1625] Output: Text and emotion information displayed on the screen

[1626] Step 7:

[1627] User (hearing impaired):

[1628] Input: Terminal input field

[1629] Action: A deaf person types a reply. Example: "Hello. The weather is nice today and I'm feeling great."

[1630] Output: Input text data

[1631] Step 8:

[1632] Terminal (hearing impaired):

[1633] Input: Text data entered by a hearing-impaired person

[1634] What it does: Sends the text and the selected emotion tag to the server.

[1635] Output: Text data and emotion information sent to the server

[1636] Step 9:

[1637] server:

[1638] Input: Received text data and emotion information

[1639] How it works: The server passes this to a TTS (Text to Speech) engine, which converts the received text data into voice data. At that time, it adds an appropriate tone to the voice based on the emotion tag. For example, for the (joy) tag, a lively and cheerful tone

[1640] Output: Audio data reflecting emotional information

[1641] Step 10:

[1642] server:

[1643] Input: Voice data reflecting emotional information

[1644] Operation: The server transmits the generated voice data to the device of the healthy person via the network.

[1645] Output: Audio data sent to the healthy subject's device

[1646] Step 11:

[1647] Device (healthy person):

[1648] Input: Transmitted audio data

[1649] Action: Plays back the audio data and has a normal subject listen to it. Example: "Hello. The weather is nice today and I'm feeling great."

[1650] Output: Played audio data

[1651] Through these steps, natural communication, including emotional content, is realized between hearing-impaired and hearing-challenged people.

[1652] (Application example 2)

[1653] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1654] It is difficult to achieve smooth and natural communication between hearing-impaired and hearing-savvy people. Conventional systems only convert voice data to text data, and lack emotional exchange in communication. This leads to insufficient communication and a decline in the quality of communication. Especially in brick-and-mortar stores, there is a strong demand for communication that includes emotional information, as this has a significant impact on customer experience and service quality.

[1655] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the received voice data into text data, emotion analysis means for analyzing the user's emotion and generating emotion information, means for assigning the generated emotion information to the text data, means for visualizing the assigned emotion information, and means for reflecting the assigned emotion information in the voice data. This enables natural and rich communication including emotion information between hearing-impaired and hearing-normal people.

[1656] "Means for receiving audio data" refers to a device or method that receives an external audio signal with low loss and converts it into a format suitable for internal processing.

[1657] A "means for converting received voice data into text data" is a device or method that uses voice recognition technology to convert received voice signals into a corresponding text format.

[1658] The "means for transmitting converted text data" refers to a device or method for transmitting the converted text data to another device or system.

[1659] The "means for receiving text data entered by a user" refers to a device or method that receives text data entered by a user and converts it into a format suitable for internal processing.

[1660] A "means for converting received text data into speech data" is a device or method that uses text-to-speech technology to convert text data into a corresponding speech format.

[1661] The "means for transmitting converted audio data" refers to a device or method for transmitting the converted audio data to another device or system.

[1662] "Emotion analysis means" refers to a device or method that reads the user's emotions from input voice or text data and generates emotion information.

[1663] The "means for adding generated emotion information to text data" refers to a device or method for adding emotion information obtained by emotion analysis to text and reinforcing the text data.

[1664] The "means for visualizing assigned emotional information" is a device or method for visualizing analyzed emotional information and presenting it to the user.

[1665] The "means for reflecting the added emotional information in the voice data" refers to a device or method for incorporating the generated emotional information into the voice synthesis process and expressing it as voice.

[1666] Overall system configuration

[1667] This system aims to realize seamless communication between hearing-impaired and hearing-suffering people. The system is broadly composed of the following elements:

[1668] Server: The emotion analysis, speech recognition, and text-to-speech synthesis engines run on a high-performance cloud server.

[1669] Terminal: A device capable of voice input / output and text display, such as a smartphone or smart glasses.

[1670] Users: Deaf and hearing impaired people who use the system.

[1671] Hardware / Software used

[1672] Server: A cloud server such as AWS EC2 or Google Cloud.

[1673] Emotion engine: Microsoft Azure Emotion API.

[1674] Speech recognition engine (ASR): Google Cloud Speech-to-Text.

[1675] Text-to-speech engine (TTS): Amazon Polly.

[1676] Program processing explanation

[1677] Voice input and emotion recognition

[1678] 1. Terminal: Voice input is performed by a healthy person speaking into the terminal.

[1679] 2. Server: The audio data captured on the device is sent to the server.

[1680] 3. Server (emotion analysis means): The received voice data is analyzed by the emotion engine, and emotional information such as joy, sadness, and anger is generated.

[1681] 4. Server (voice recognition means): Voice data is converted into text data by the ASR engine.

[1682] 5. Server (means for adding emotional information): Emotional information is added to text data as tags. For example, "Hello. How are you today? (Joy)".

[1683] Displaying text data and conveying emotions

[1684] 6. Server: The generated emotion-tagged text data is sent to the hearing-impaired person's device.

[1685] 7. Terminal (visualization means): The text data is decoded and displayed on the screen, and emotion tags are visually displayed using icons and colors.

[1686] Text input and emotion communication for the hearing impaired

[1687] 8. Device: A hearing impaired person uses a device to input text, for example, "Hello. The weather is nice today and I feel good. (Joy)."

[1688] 9. Server: The input text data is sent to the server along with the emotion information.

[1689] Text-to-speech and emotional response

[1690] 10. Server (text-to-speech synthesis means): The received text data is passed to the TTS engine, which analyzes the text data and generates voice data.

[1691] 11. Server (Means for Reflecting Emotional Information): The server adds intonation to the voice data based on the emotional information. For example, if the voice data is tagged with "joy," it generates a voice with a lively and cheerful tone.

[1692] 12. Terminal: The generated voice data is played on the terminal of the able-bodied person.

[1693] Examples of concrete examples and prompts

[1694] Here is a concrete example of the dialogue:

[1695] 1. Healthy person: "Hello. How are you today?" (joy).

[1696] 2. Hearing impaired person: "Hello. The weather is nice today and I feel great." (joy).

[1697] Example prompt for a generative AI model:

[1698] "The store clerk greeted: Hello, is there anything you're looking for?"

[1699] "Customer enters through smart glasses: I'm not looking for anything in particular today."

[1700] "Convert these messages into text and audio with appropriate emotional information."

[1701] This system enables natural and rich communication, including emotional information, between hearing-impaired and hearing-savvy people. Specifically, it is expected to improve the customer experience and service quality in brick-and-mortar stores.

[1702] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1703] Step 1:

[1704] Device: Voice input is performed by a non-disabled person speaking into the device. The device's built-in microphone captures the voice signal and stores it as digitally formatted voice data. This voice data is sent to a server for further processing. The input here is the voice spoken by a non-disabled person, and the output is digitally formatted voice data.

[1705] Step 2:

[1706] Server: Receives the voice data sent from the device. The server checks the format of the voice data and converts it to the appropriate format if necessary. At this stage, data pre-processing is performed, such as noise filtering. The input is voice data, and the output is pre-processed voice data.

[1707] Step 3:

[1708] Server (emotion analysis means): Analyzes the received voice data using an emotion engine to generate emotional information for healthy individuals. In the analysis step, emotions are estimated based on the pitch, tone, and speed of the voice. The input is processed voice data, and the output is metadata containing emotional information.

[1709] Step 4:

[1710] Server (voice recognition means): After emotion analysis, the voice data is converted into text data by the ASR engine. The voice recognition engine analyzes the voice waveform and converts it into the corresponding text format. The input here is the voice data that has been emotion analyzed, and the output is the converted text data.

[1711] Step 5:

[1712] Server (emotion information adding means): Adds the generated emotion information to the converted text data as a tag. Specifically, emotion information is added to the end of a sentence in the text. For example, "Hello. How are you today? (Joy)". The input is text data and emotion metadata, and the output is emotion-tagged text data.

[1713] Step 6:

[1714] Server: Sends emotion-tagged text data to the device of the hearing-impaired person. Data transfer takes place here, and stable communication is ensured using network protocols (e.g., HTTP, WebSocket). The input is emotion-tagged text data, and the output is the text data received by the device of the hearing-impaired person.

[1715] Step 7:

[1716] Terminal (Visualization Means): The terminal of the hearing-impaired person decodes the received text data and displays it on the screen. At the same time, visual feedback (e.g., icons or colors) based on the emotion tag is also displayed. The input is emotion-tagged text data, and the output is the displayed text and emotion information.

[1717] Step 8:

[1718] User: A hearing-impaired person uses the device to input a text message. For example, "Hello. The weather is nice today and I feel great. (Joy)." The device receives the user's input and stores it as text data. The input is the text entered by the user, and the output is text data.

[1719] Step 9:

[1720] Terminal: Sends input text data to the server. The terminal's communication module sends the text data to the server, and the data is encrypted if necessary. The input is text data, and the output is the text data sent to the server.

[1721] Step 10:

[1722] Server (Text-to-Speech Synthesis Means): The received text data is converted into voice data by the TTS engine. The TTS engine analyzes the text and generates natural-sounding voice. The input is text data, and the output is the generated voice data.

[1723] Step 11:

[1724] Server (emotional information reflection means): Reflects emotional information in the generated voice data. Specifically, it adjusts the tone and pitch of the voice based on the emotion tag. The input is voice data and emotional information, and the output is voice data that reflects the emotional information.

[1725] Step 12:

[1726] Terminal: The audio data reflecting emotional information is played on the terminal of the hearing-impaired person. Here, the hearing-impaired person can receive a message from the hearing-impaired person by voice. The input is the audio data reflecting emotional information, and the output is the audio played from the terminal.

[1727] In this way, the system can realize natural communication including emotional information between hearing-impaired and hearing-normal people.

[1728] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1729] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1730] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1731] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1732] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1733] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1734] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1735] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1736] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1737] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1738] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1739] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1740] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1741] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1742] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1743] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1744] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1745] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1746] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1747] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1748] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1749] The following is further disclosed regarding the above embodiment.

[1750] (Claim 1)

[1751] means for receiving audio data;

[1752] means for converting received voice data into text data;

[1753] means for transmitting the converted text data;

[1754] means for receiving text data entered by a user;

[1755] means for converting received text data into voice data;

[1756] means for transmitting the converted audio data;

[1757] A system including:

[1758] (Claim 2)

[1759] 2. The system according to claim 1, wherein the means for receiving voice data acquires the voice data from an input device of a healthy person.

[1760] (Claim 3)

[1761] 2. The system according to claim 1, wherein the means for receiving the text data input by the user acquires the text data from an input device of a hearing impaired person.

[1762] "Example 1"

[1763] (Claim 1)

[1764] means for receiving audio data;

[1765] means for converting received voice data into text data;

[1766] means for transmitting the converted text data;

[1767] means for receiving text data entered by a user;

[1768] means for converting received text data into voice data;

[1769] means for transmitting the converted audio data;

[1770] a means including an automatic speech recognition engine using a generative AI model;

[1771] a means including a text-to-speech engine using a generative AI model;

[1772] A system including:

[1773] (Claim 2)

[1774] 2. The system according to claim 1, wherein the means for receiving voice data acquires the voice data from an input device of a healthy person.

[1775] (Claim 3)

[1776] 2. The system according to claim 1, wherein the means for receiving the text data input by the user acquires the text data from an input device of a hearing impaired person.

[1777] "Application Example 1"

[1778] (Claim 1)

[1779] means for receiving audio data;

[1780] means for converting received voice data into text data;

[1781] means for transmitting the converted text data;

[1782] means for receiving text data entered by a user;

[1783] means for converting received text data into voice data;

[1784] means for transmitting the converted audio data;

[1785] A means for enabling two-way communication of text data and voice data between a customer and a store clerk in a physical store;

[1786] A system including:

[1787] (Claim 2)

[1788] 2. The system according to claim 1, wherein the means for receiving voice data acquires the voice data from an input device of a healthy person.

[1789] (Claim 3)

[1790] 2. The system according to claim 1, wherein the means for receiving the text data input by the user acquires the text data from an input device of a hearing impaired person.

[1791] "Example 2: Combining Emotion Engines"

[1792] (Claim 1)

[1793] means for receiving audio data;

[1794] means for analyzing emotional information from the received voice data;

[1795] means for converting the voice data to which emotion information has been added into text data;

[1796] means for transmitting the converted text data and emotion information;

[1797] means for receiving text data input by a user together with emotion information;

[1798] means for converting the received text data and emotion information into voice data;

[1799] means for transmitting the converted voice data and emotion information;

[1800] A system including:

[1801] (Claim 2)

[1802] 2. The system according to claim 1, wherein the means for receiving voice data acquires the voice data from an input device of a healthy person.

[1803] (Claim 3)

[1804] 2. The system according to claim 1, wherein the means for receiving the text data and emotion information input by the user acquires the text data and emotion information from an input device of a hearing impaired person.

[1805] "Application example 2 when combining emotion engines"

[1806] Claiming a new invention

[1807] (Claim 1)

[1808] means for receiving audio data;

[1809] means for converting received voice data into text data;

[1810] means for transmitting the converted text data;

[1811] means for receiving text data entered by a user;

[1812] means for converting received text data into voice data;

[1813] means for transmitting the converted audio data;

[1814] emotion analysis means for analyzing a user's emotion and generating emotion information;

[1815] means for adding the generated emotion information to the text data;

[1816] A means for visualizing the added emotional information;

[1817] a means for reflecting the added emotional information in the voice data;

[1818] A system including:

[1819] (Claim 2)

[1820] 2. The system according to claim 1, wherein the means for receiving voice data acquires the voice data from an input device of a healthy person.

[1821] (Claim 3)

[1822] 2. The system according to claim 1, wherein the means for receiving the text data input by the user acquires the text data from an input device of a hearing impaired person. [Explanation of symbols]

[1823] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving audio data; means for converting received voice data into text data; means for transmitting the converted text data; means for receiving text data entered by a user; means for converting received text data into voice data; means for transmitting the converted audio data; A system including:

2. 2. The system according to claim 1, wherein the means for receiving voice data acquires voice data from an input device of a healthy person.

3. 2. The system of claim 1, wherein the means for receiving the user-inputted text data acquires the text data from an input device of a hearing-impaired person.

Images