System

Abstract

A system is provided.SOLUTION: A system comprising: first means for acquiring a voice input from a user; second means for converting the voice input into text data; third means for transmitting the text data to a natural language processing model to generate a response text; fourth means for converting the response text into voice data; fifth means for playing the voice data to the user; sixth means for inputting the text data, transmitting the text data to the natural language processing model, and receiving a response; and seventh means for receiving the response text generated by the sixth means, converting the response text into voice data, and playing the voice data to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, communication between the hearing-impaired and the visually impaired, or between people with and without disabilities, is difficult. In particular, the exchange of information between the visually impaired, who require audio information, and the hearing-impaired, who require text information, is not smooth, which creates constraints when working or studying together in the workplace or school. This makes it difficult for people with disabilities to communicate equally, resulting in social isolation. There is a need for a means to solve these problems and enable everyone to share information equally and communicate smoothly. [Means for solving the problem]

[0005] The present invention solves these problems with a system that includes a means for converting speech input into text data, a means for transmitting the text data to a natural language processing model to generate a response text, a means for converting the response text into voice data, a means for playing the voice data to a user, a means for inputting text data and transmitting it to a natural language processing model to receive a response, and a means for receiving the generated response text, converting it into voice data, and playing it back to a user. Specifically, the system converts speech into text using voice recognition technology, generates an appropriate response using natural language processing, and further converts text into voice using voice synthesis technology, thereby enabling smooth communication between users who require information in different formats. This makes it possible to provide an environment in which people with and without disabilities can thrive in the same workplace or school.

[0006] "Audio Input" is audio data captured using a microphone or other audio capture device.

[0007] "Text data" is character string information obtained by analyzing voice input.

[0008] "Natural language processing model" refers to machine learning algorithms and their behavior for understanding and generating human language.

[0009] "Response text" is a response string generated by a natural language processing model.

[0010] "Voice data" refers to information in audio format that is generated using voice synthesis technology.

[0011] "Voice recognition technology" is a technology for converting voice input into text data.

[0012] "Speech synthesis technology" is a technology for converting text data into voice data.

[0013] "User" refers to a blind, hearing impaired, or other user who operates the system.

[0014] A "terminal" is a device that controls voice input and output, text input and output. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of the present invention converts voice input into text data, generates an appropriate response using a natural language processing model, and then converts that response back into voice data, thereby enabling smooth communication between visually impaired and hearing impaired people, or between people with and without disabilities.

[0037] In this system, the terminal is equipped with a microphone to capture voice input from the user, and the captured voice data is transmitted to the server in real time. The server then uses voice recognition technology to convert the voice input into text data.

[0038] The server then sends the converted text data to a natural language processing model (e.g., a large-scale language model) to generate an appropriate response text, which is then processed again within the server and converted into voice data using speech synthesis technology.

[0039] The terminal then plays the converted audio data back to the user, allowing visually and hearing impaired people to receive information in a format that is convenient for them.

[0040] As a specific example, consider a scenario in which visually impaired user A says, "Hello, how are you?"

[0041] 1. The device captures user A's voice with a microphone and sends the voice data to the server.

[0042] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[0043] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you."

[0044] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[0045] 5. The terminal plays the audio data to User A.

[0046] On the other hand, consider the case where hearing-impaired user B inputs and receives information in text format.

[0047] 1. User B enters "It's a nice day today" in text format.

[0048] 2. The terminal sends the entered text data to the server.

[0049] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[0050] 4. The server sends the response text to the terminal, which displays it to User B.

[0051] This allows the visually impaired to communicate through speech, and the hearing impaired to communicate through text. By utilizing speech recognition, natural language processing, and speech synthesis technologies at each stage, instant and smooth information exchange becomes possible. This invention enables people with and without disabilities to work and study in the same workplace or school, realizing a more inclusive and equal society.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[0055] Step 2:

[0056] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[0057] Step 3:

[0058] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[0059] Step 4:

[0060] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[0061] Step 5:

[0062] The server sends the converted text data to a natural language processing model to generate an appropriate response text. For example, a natural language processing model can be used to generate a response such as "Yes, I'm fine, thanks to you."

[0063] Step 6:

[0064] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[0065] Step 7:

[0066] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[0067] Step 8:

[0068] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[0069] Step 9:

[0070] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[0071] Step 10:

[0072] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[0073] Step 11:

[0074] The server sends the received text data to a natural language processing model to generate an appropriate response text, such as "Yes, I feel like going for a walk."

[0075] Step 12:

[0076] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[0077] Step 13:

[0078] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[0079] This series of processes realizes a system that allows smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] The problem that this invention aims to solve is to realize smooth, real-time communication between users with different disabilities, such as the visually or hearing impaired, and between users without disabilities. In particular, the object is to provide a system that integrates voice input, voice recognition, natural language processing, and voice synthesis technologies, allowing users to receive and send information in a format that is suitable for them.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text and playing it back to the user, thereby enabling smooth communication between the visually impaired and the hearing impaired, and between users with no disabilities.

[0085] "User" refers to a person who uses the system to input voice or text and receives a response.

[0086] "Audio input" refers to audio signals provided by a user to a system through a microphone device.

[0087] "Text data" refers to text information generated by voice recognition technology or text input.

[0088] "Natural language processing model" refers to a machine learning algorithm for generating appropriate response text from text data.

[0089] "Response text" is the response content generated by the natural language processing model expressed as text information.

[0090] "Voice data" refers to a voice signal converted from response text using voice synthesis technology.

[0091] "Speech recognition technology" refers to technology for converting voice data into text data.

[0092] "Speech synthesis technology" refers to technology for converting text data into natural-sounding speech.

[0093] "Microphone device" refers to a device for capturing a user's voice input.

[0094] A "generative AI model" refers to an algorithm that performs natural language processing using advanced machine learning techniques such as large-scale language models.

[0095] "Communication protocol" refers to the rules and procedures used to send and receive data.

[0096] The present invention is a system that enables smooth communication between visually impaired and hearing impaired people, or between users with and without disabilities, by converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data.

[0097] The device is equipped with a microphone device to capture voice input from the user. When the user speaks, the device's microphone converts the voice into digital audio data. The captured audio data is transmitted to the server in real time. This data transmission uses a secure and fast communication protocol (e.g., HTTPS or WebSocket).

[0098] The server converts the voice data into text data using speech recognition technology. Specifically, it uses a speech recognition engine (e.g., a general speech recognition API). The converted text data is sent to a natural language processing model (e.g., a large-scale language model). The natural language processing model generates an appropriate response text from the text data.

[0099] The generated response text is converted back into audio data using speech synthesis technology. A speech synthesis engine (for example, a general speech synthesis API) is used. This audio data is then sent back to the device and played back to the user. This allows visually and hearing impaired people to receive information in a format that is suitable for them.

[0100] Specific examples

[0101] Scenario in which visually impaired user A speaks

[0102] 1. The device captures user A's voice saying "Hello, how are you?" and sends the voice data to the server.

[0103] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[0104] 3. The server sends the converted text data to a natural language processing model and generates the response text "Yes, I'm fine, thanks to you."

[0105] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[0106] 5. The terminal plays the converted voice data to User A.

[0107] Scenario where hearing impaired user B inputs information in text format

[0108] 1. User B enters "It's a nice day today" in text format.

[0109] 2. The terminal sends the entered text data to the server.

[0110] 3. The server sends the input text data to a natural language processing model and generates the response text "Yes, I feel like going for a walk."

[0111] 4. The server sends the generated response text to the terminal, which displays it to User B.

[0112] Prompt Sentence Examples

[0113] Case of visually impaired user A:

[0114] User A says "Hello, how are you?" Generate an appropriate response to this.

[0115] Deaf User B's Case:

[0116] User B has entered the text "It's a nice day today." Generate an appropriate response to this.

[0117] This system is designed to enable smooth communication between visually and hearing impaired people, as well as users with no disabilities, and makes use of voice recognition, natural language processing, and voice synthesis technologies at each stage to enable instant and smooth information exchange.

[0118] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0119] Step 1:

[0120] The device uses a microphone device to capture voice input from the user. When the user says "Hello, how are you?", the device converts the voice into digital voice data. The input is the user's voice, and the output is digital voice data. This voice data is saved in a format such as PCM.

[0121] Step 2:

[0122] The device transmits the captured audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket). The input is the captured digital audio data, and the output is the data sent to the server.

[0123] Step 3:

[0124] The server converts the received voice data into text data using voice recognition technology. Specifically, the conversion is performed using a voice recognition engine. The input is digital voice data, and the output is text data such as "Hello, how are you?" The voice recognition engine analyzes the voice data and converts it into text information.

[0125] Step 4:

[0126] The server sends the converted text data to a natural language processing model. Specifically, it uses a large-scale language model to generate an appropriate response text from the text data. The input is the text data "Hello, how are you?", and the output is the response text "Yes, thanks to you, I'm fine."

[0127] Step 5:

[0128] The server converts the generated response text into voice data. It uses voice synthesis technology to convert text data into voice. Specifically, it uses a voice synthesis engine. The input is the response text "Yes, I'm fine, thanks to you," and the output is digital voice data.

[0129] Step 6:

[0130] The server transmits the converted voice data to the terminal using a secure communication protocol. The input is digital voice data, and the output is data to be transmitted to the terminal.

[0131] Step 7:

[0132] The terminal plays the received audio data, and the user can listen to the audio through the speaker. The input is the digital audio data sent from the server, and the output is the audio played to the user.

[0133] This series of processes enables smooth communication between the visually impaired, the hearing impaired, and users without disabilities. Specifically, the process combines appropriate technologies for each step: voice capture, data transmission, voice recognition, natural language processing, voice synthesis, and voice playback.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Visually and hearing impaired people face challenges in communicating smoothly in virtual stores. In particular, they face difficulties in sharing information with store clerks and other customers in a timely manner, which often leads to stress and inconvenience in the process of obtaining specific information or services. Furthermore, conventional systems do not adequately achieve real-time voice input, text conversion, or appropriate response generation.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes means for acquiring voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for supporting communication between customers and shop staff in a virtual store, means for receiving voice input from the user in real time via a smart device and generating an appropriate response, and means for presenting the response generated by the above means to the user. This enables visually impaired and hearing impaired people to communicate smoothly and in real time using voice and text in a virtual store.

[0139] "User" refers to a person or entity that uses the System.

[0140] "Means for obtaining voice input" refers to the technology or device that captures the voice uttered by the user and that the system captures the voice data.

[0141] "Means for converting into text data" refers to a technology or device for converting acquired voice data into text information.

[0142] "Natural language processing model" refers to a machine learning model used to analyze input text data and generate an appropriate response.

[0143] "Response text" refers to text data of a response generated by a natural language processing model.

[0144] "Means for converting into voice data" refers to a technology or device for converting response text into voice data.

[0145] "Means for playing audio data to a user" refers to a technique or device for letting a user hear the generated audio data.

[0146] A "virtual store" refers to a virtual shopping or service space that is developed online.

[0147] "Customer" refers to a user who uses the virtual store.

[0148] "Shop staff" refers to a person or AI avatar set up to interact with customers within a virtual store.

[0149] "Smart device" refers to a device (e.g., smartphone, smart glasses, etc.) that has internet connectivity and can use a wide variety of applications.

[0150] "Means for generating an appropriate response" refers to a technique or device for generating a response to a user's input using a natural language processing model.

[0151] "Means for presenting to the user" refers to a technology or device that visually or audibly displays to the user the response generated through the smart device.

[0152] This invention is a system that supports smooth communication within a virtual store for visually and hearing impaired people. This system allows users to exchange information within the store via voice and text through their own smart devices.

[0153] 1. Program Overview

[0154] The system includes the following elements:

[0155] Acquire voice input: Acquire voice input from the user.

[0156] Speech recognition: Converts voice input into text data.

[0157] Natural Language Processing: Sending text data to a natural language processing model to generate a response.

[0158] Speech synthesis: Converts response text into audio data.

[0159] Audio playback: The generated audio data is played back to the user.

[0160] Text input and response reception: Send text data to the natural language processing model and receive a response.

[0161] Communication support in virtual stores: Supports communication between customers and store staff within virtual stores.

[0162] 2. System configuration

[0163] Voice input acquisition: Voice input is acquired through the microphone of the smart device (smart glasses or smartphone).

[0164] Speech recognition: Uses the Google Speech Recognition API to convert voice data into text data.

[0165] Natural Language Processing: Uses a Transformer model (e.g., rinna / japanese-gpt-1b) to generate appropriate responses to input text.

[0166] Speech synthesis: Use gTTS (Google Text-to-Speech) to convert the generated response text into voice data.

[0167] Audio playback: Use Pygame to play audio data and present it to the user.

[0168] Text input and response reception: The system sends text input to a natural language processing model and receives the generated response, which it displays to the user.

[0169] Communication support in virtual stores: Receives voice input from users in real time via smart devices and generates and presents appropriate responses.

[0170] 3. Specific examples of processing

[0171] As a concrete example, consider a visually impaired person looking for product information in a virtual store:

[0172] 1. A visually impaired person asks the smart glasses, "Tell me about this product."

[0173] 2. The microphone in the smart glasses captures the audio and converts it to text via the Google Speech Recognition API.

[0174] 3. The text data is sent to a natural language processing model (rinna / japanese-gpt-1b), which generates the response, "This product is a state-of-the-art smartphone. Its features include a high-resolution camera and a large-capacity battery."

[0175] 4. The generated response text is converted to speech using gTTS.

[0176] 5. The audio data is played to the user using Pygame, and the user receives information through audio.

[0177] Example prompt sentence:

[0178] "Prompt: User has asked a question about a product. Please provide a brief description of the relevant product information."

[0179] In this way, the system embodying the present invention enables visually impaired and hearing impaired people to smoothly obtain information and communicate within a virtual store.

[0180] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0181] Step 1:

[0182] The device captures voice input from the user using a microphone. The input is voice data and prepares it to be sent to a speech recognition API. Specifically, the microphone captures the user's voice, converts the data into a digital format, and stores it in memory.

[0183] Step 2:

[0184] The device sends the captured voice data to the Google Speech Recognition API. The input is digital voice data, and the output is corresponding text data. Specifically, the digital voice data is sent to the API endpoint, which performs real-time speech-to-text conversion.

[0185] Step 3:

[0186] The server receives text data from the Google Speech Recognition API and sends it to the natural language processing model. The input is the converted text data, and the output is the generated response text. Specifically, the text data is sent to the API of the natural language processing model, which generates appropriate text as a response.

[0187] Step 4:

[0188] The server sends the generated response text to the Google Text-to-Speech (gTTS) API and converts it into voice data. The input is the response text and the output is voice data. Specifically, the response text is sent to the gTTS API, which uses speech synthesis technology to convert the text into voice.

[0189] Step 5:

[0190] The device plays the generated audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to play the audio data, and the audio is output through the device's speaker.

[0191] Step 6:

[0192] The user inputs text data arbitrarily. The input is text data, which is then prepared for transmission to the natural language processing model. Specifically, the text data is input using the device's keyboard or voice recognition function.

[0193] Step 7:

[0194] The server sends the input text data to the natural language processing model and receives a response. The input is text data and the output is response text. Specifically, the server sends text data to the natural language processing model and receives appropriate text as a response.

[0195] Step 8:

[0196] The server converts the received response text into voice data. The input is the text data to be converted, and the output is voice data. Specifically, the server uses the gTTS API to convert text data into voice data.

[0197] Step 9:

[0198] The device plays the converted audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to output the audio through the device's speakers.

[0199] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0200] This invention is a communication support system that combines an emotion engine, which converts voice input into text data, generates an appropriate response using a natural language processing model, and converts that response into voice data, thereby realizing smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities. Furthermore, by using the emotion engine, emotions can be added to the user's utterances, further supporting natural communication.

[0201] In this system, the terminal is equipped with a microphone for capturing voice input from the user and acquires the voice input. The captured voice data is sent to a server in real time. The server uses voice recognition technology to convert the voice input into text data. The converted text data is then sent to an emotion engine, which recognizes the user's emotion. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), which generates a response text. The generated response text is again processed within the server and converted into voice data using voice synthesis technology. The terminal plays the converted voice data to the user. In this way, visually and hearing impaired people can receive information in a format that is suitable for them.

[0202] As a specific example, suppose visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data, "Hello, how are you?" The emotion engine then recognizes user A's emotion from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text, "Yes, I'm fine, thanks to you." At this time, a response is generated that includes the "friendly" emotion. The server then converts the response text into voice data and sends the voice data to the device. The device plays the voice data to user A.

[0203] Meanwhile, consider the case where hearing-impaired User B responds by text input. User B inputs "It's a nice day today, isn't it?" in text format. The device sends the input text data to the server. The server sends the text data to the emotion engine, which recognizes the emotion "joy" from User B's input. The server sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk." The server then sends the response text back to the device in text format. The device displays the received text data to User B.

[0204] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.The present invention aims to improve the quality of communication, enable people with disabilities to share information equally, and reduce social isolation.

[0205] The processing flow will be explained below.

[0206] Step 1:

[0207] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[0208] Step 2:

[0209] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[0210] Step 3:

[0211] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[0212] Step 4:

[0213] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[0214] Step 5:

[0215] The server sends the converted text data to an emotion engine to recognize the user's emotion, for example, using the emotion engine to recognize the "friendly" emotion.

[0216] Step 6:

[0217] The server sends the text data and the recognized emotion data to a natural language processing model to generate an appropriate response text. For example, the natural language processing model generates a response such as "Yes, thanks to you, I'm fine."

[0218] Step 7:

[0219] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[0220] Step 8:

[0221] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[0222] Step 9:

[0223] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[0224] Step 10:

[0225] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[0226] Step 11:

[0227] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[0228] Step 12:

[0229] The server sends the received text data to the emotion engine and recognizes the emotion of User B. For example, it recognizes the emotion of "joy."

[0230] Step 13:

[0231] The server sends the text data and recognized emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[0232] Step 14:

[0233] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[0234] Step 15:

[0235] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[0236] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.

[0237] Example 2

[0238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] The present invention aims to enable smooth communication between users who have communication difficulties, such as the visually or hearing impaired. In particular, by combining technologies of speech recognition, emotion recognition, natural language processing, and speech synthesis, the system generates natural responses by adding emotion to user utterances, supporting two-way communication. Another objective is to provide a system that allows information to be transmitted seamlessly even when users use different input methods (voice or text).

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0241] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to an emotion recognition engine and acquiring emotion data, means for transmitting the emotion data and the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text generated by the seventh means, converting it into voice data, and playing it back to the user. This enables visually impaired and hearing impaired people to communicate naturally with emotions.

[0242] "Voice input" is the act or means of obtaining a user's spoken words as a digital signal.

[0243] "Text data" is a data format in which voice input is converted into text information.

[0244] An "emotion recognition engine" is software or a system for analyzing and identifying user emotions from text data.

[0245] A "natural language processing model" is a machine learning model that analyzes given text data and generates responses based on grammar and context.

[0246] "Response text" is text data generated in response to a user input.

[0247] "Audio data" is a data format in which text data is converted into a digital audio signal.

[0248] "Voice recognition technology" is a technology for converting voice input into text data.

[0249] "Speech synthesis technology" is a technology for converting text data into voice data.

[0250] "User" refers to an individual or entity that uses this system, and includes the visually impaired and the hearing impaired.

[0251] This invention is a communication support system that incorporates an emotion recognition engine. By converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data, it enables smooth communication between the visually impaired, the hearing impaired, and people with and without disabilities. Furthermore, by using an emotion recognition engine, it is possible to assign emotions to the user's speech, supporting more natural communication.

[0252] This system consists of the following main hardware and software components: First, the device is equipped with a microphone that captures voice input from the user. This voice data is sent to a server in real time. The server converts the voice data into text data using voice recognition technology (e.g., a voice recognition API). Next, the converted text data is sent to an emotion recognition engine (e.g., an emotion recognition API), which analyzes the user's emotions. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), and a response text is generated. The generated response text is again processed within the server and converted into voice data using voice synthesis technology (e.g., a voice synthesis API). The device then plays the converted voice data to the user.

[0253] As a specific example, consider the case where visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?" The emotion recognition engine then recognizes user A's emotions from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you." The "friendly" emotion is also taken into consideration at this time. The server then converts the response text into voice data and sends the voice data to the device. The device then plays the voice data to user A.

[0254] On the other hand, consider the case where hearing-impaired user B inputs "It's a nice day today." User B inputs in text format, and the device sends the text data to the server. The server sends the text data to an emotion recognition engine, which recognizes the emotion "joy" from user B's input. The server sends the text data and emotion data to a natural language processing model, which generates a response text: "Yes, I feel like going for a walk." The server then sends the generated response text back to the device in text format, and the device displays the received text data to user B.

[0255] Example prompt sentence:

[0256] 1. "When visually impaired person A says, 'Hello, how are you?' Please generate a response text. Please make it a little friendly."

[0257] 2. "Generate a response to the user's input 'What a nice day today!' with the emotion 'Joy'."

[0258] This series of processes enables all users, including those with visual or hearing impairments, to communicate naturally and with emotion.

[0259] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0260] Step 1:

[0261] The user speaks into the microphone, for example, "Hello, how are you?" This is the voice input to the system.

[0262] Input: Speech data ("Hello, how are you?")

[0263] Output: Captured audio data

[0264] Step 2:

[0265] The device uses a microphone to capture the user's voice and obtains the voice data in real time.

[0266] Input: Speech data ("Hello, how are you?")

[0267] Output: Digital audio data

[0268] Step 3:

[0269] The device then transmits the captured audio data over the internet to a server, which is often encrypted to ensure secure transmission.

[0270] Input: Digital audio data

[0271] Output: Audio data sent to the server

[0272] Step 4:

[0273] The server converts the voice data into text data using voice recognition technology (for example, a voice recognition API), which recognizes the voice as text.

[0274] Input: Audio data

[0275] Output: Text data ("Hello, how are you?")

[0276] Step 5:

[0277] The server sends the text data to an emotion recognition engine (e.g., an emotion recognition API) to analyze the user's emotion.

[0278] Input: Text data ("Hello, how are you?")

[0279] Output: Emotion data ("Friendly")

[0280] Step 6:

[0281] The server sends the text data and emotion data to a natural language processing model (e.g., a large-scale language model) to generate a response text.

[0282] Input: Text data ("Hello, how are you?"), Emotion data ("Friendly")

[0283] Output: Response text ("Yes, I'm fine, thanks to you.")

[0284] Step 7:

[0285] The server converts the generated response text into voice data using a voice synthesis technology (for example, a voice synthesis API).

[0286] Input: Response text ("Yes, I'm fine, thanks to you.")

[0287] Output: Audio data

[0288] Step 8:

[0289] The server then sends the audio data back to the device over the internet, where it is again transferred securely.

[0290] Input: Audio data

[0291] Output: Audio data transferred to the device

[0292] Step 9:

[0293] The terminal plays the received audio data over a speaker to provide feedback to the user.

[0294] Input: Audio data

[0295] Output: Played speech ("Yes, I'm fine, thanks to you.")

[0296] Through the above steps, all users, including those with visual or hearing impairments, can communicate naturally and with emotion.

[0297] (Application example 2)

[0298] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] Visually and hearing impaired people face challenges when communicating with others in public places or stores, such as difficulty sending and receiving information smoothly. They also face difficulties in communicating naturally and emotionally, which can lead to a sense of social isolation. Therefore, there is a need for a new system that allows people with disabilities to share information equally and realize natural and smooth dialogue.

[0300] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0301] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for recognizing emotions from the text data using an emotion engine, means for generating the response text to which the emotions have been added, and means for converting the response text into voice data, thereby enabling visually impaired and hearing impaired people to communicate naturally and smoothly in public places such as brick-and-mortar stores.

[0302] "User" refers to a person who uses the system.

[0303] "Voice input" refers to voice data collected from a user's speech through an input device such as a microphone.

[0304] "Voice recognition technology" refers to technology that analyzes voice data and converts its content into text data.

[0305] "Text data" refers to data that has been converted from voice input into a string of characters using voice recognition technology.

[0306] A "natural language processing model" refers to an artificial intelligence model that analyzes text data, understands its meaning, and generates an appropriate response.

[0307] "Response Text" refers to the response string generated by a natural language processing model.

[0308] An "emotion engine" refers to technology that recognizes a user's emotions from text data and analyzes those emotions.

[0309] "Audio data" refers to data that has been converted back into audio format from text data using speech synthesis technology.

[0310] "Speech synthesis technology" refers to technology for outputting text data as voice data.

[0311] "Playing back to the user" means letting the user hear the synthesized voice data via a speaker, earphones, or the like.

[0312] "System" refers to a collection of devices and software that performs a series of processes including voice input acquisition, speech recognition, natural language processing, emotion recognition, and speech synthesis.

[0313] As an embodiment of the present invention, we will explain a system that enables visually impaired and hearing impaired people to communicate smoothly in physical stores. This system uses smart glasses and related terminals owned by users, and a server.

[0314] First, the user inputs voice through the smart glasses. The smart glasses have a built-in microphone that captures the user's voice. The captured voice data is sent to the server in real time. The server then converts the voice data into text data using the speech_recognition library.

[0315] Next, this text data is sent to a natural language processing model, which is a generative AI model. The server also contains an emotion engine that analyzes the user's emotions from the received text data. The emotion engine recognizes emotions such as "friendly" or "joyful" and adds this emotional information to the natural language processing model.

[0316] A natural language processing model takes emotion into account to generate an appropriate response text. This response is then converted into audio data on the server and sent to the smart glasses, which then play the response aloud to the user using a bone conduction speaker.

[0317] As a concrete example, consider the case where a user wears smart glasses and says, "Excuse me, where can I find the bread?" while in a store. The smart glasses capture the voice and send it to the server. The server performs speech recognition and converts it into text data, generating the text data "Excuse me, where can I find the bread." This text data is then sent to a natural language processing model by an emotion engine along with a "friendly" emotion. The response generated is "It's near the refrigerated section. Can I help you? You're very welcome." Finally, the smart glasses play this response to the user through a bone conduction speaker.

[0318] Example prompt sentence:

[0319] The user says, "Excuse me, where is the bread?" Please respond with a friendly sentiment.

[0320] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0321] Step 1:

[0322] The user inputs voice through the smart glasses. The microphone built into the smart glasses captures the user's voice. The input is the user's voice data, and the output is the captured voice data. A specific example of this is when the user says, "Excuse me, where is the bread?"

[0323] Step 2:

[0324] The captured audio data is sent to the server in real time. The input is the audio data captured by the smart glasses, and the output is the audio data sent to the server. Specifically, the smart glasses transfer the audio data to the server via Wi-Fi or Bluetooth.

[0325] Step 3:

[0326] The server converts the voice data into text data using the speech_recognition library. The input is the voice data sent to the server, and the output is the text data "Excuse me, where is the bread?". Specifically, the server receives the voice data and executes the process of converting it into text.

[0327] Step 4:

[0328] The server sends this text data to a natural language processing model, which is a generative AI model. The input is the converted text data, and the output is the transmission of the text data to the natural language processing model. Specifically, the server uses an API to send the text data to the model.

[0329] Step 5:

[0330] The emotion engine of the server recognizes the user's emotion from the text data. The input is the text data sent to the natural language processing model, and the output is the emotion recognition data "friendly." In specific operations, the emotion engine of the server analyzes the text data and identifies the emotion.

[0331] Step 6:

[0332] The natural language processing model takes emotion into account and generates a response text. The input is the emotion recognition data "friendly" and the text data "Excuse me, where is the bread?", and the output is the response text "It's near the refrigerated section. Can I help you? You're most welcome." In concrete terms, the natural language processing model takes emotion into account and generates an appropriate response.

[0333] Step 7:

[0334] The server converts the generated response text into voice data. The input is the response text and the output is voice data. Specifically, the server uses voice synthesis technology to convert the text data into voice data.

[0335] Step 8:

[0336] The converted voice data is sent to the smart glasses. The input is the voice data generated by the server, and the output is the voice data sent to the smart glasses. Specifically, the server transfers the voice data to the smart glasses again via Wi-Fi or Bluetooth.

[0337] Step 9:

[0338] The smart glasses use a bone conduction speaker to play audio data to the user. The input is the audio data sent to the smart glasses, and the output is the audio heard by the user. Specifically, the bone conduction speaker of the smart glasses plays the audio to the user, saying, "You're near the refrigerated section. Can you help me? You're very welcome."

[0339] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0340] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0341] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0342] [Second embodiment]

[0343] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0344] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0345] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0346] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0347] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0348] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0349] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0350] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0351] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0352] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0353] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0354] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0355] The system of the present invention converts voice input into text data, generates an appropriate response using a natural language processing model, and then converts that response back into voice data, thereby enabling smooth communication between visually impaired and hearing impaired people, or between people with and without disabilities.

[0356] In this system, the terminal is equipped with a microphone to capture voice input from the user, and the captured voice data is transmitted to the server in real time. The server then uses voice recognition technology to convert the voice input into text data.

[0357] The server then sends the converted text data to a natural language processing model (e.g., a large-scale language model) to generate an appropriate response text, which is then processed again within the server and converted into voice data using speech synthesis technology.

[0358] The terminal then plays the converted audio data back to the user, allowing visually and hearing impaired people to receive information in a format that is convenient for them.

[0359] As a specific example, consider a scenario in which visually impaired user A says, "Hello, how are you?"

[0360] 1. The device captures user A's voice with a microphone and sends the voice data to the server.

[0361] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[0362] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you."

[0363] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[0364] 5. The terminal plays the audio data to User A.

[0365] On the other hand, consider the case where hearing-impaired user B inputs and receives information in text format.

[0366] 1. User B enters "It's a nice day today" in text format.

[0367] 2. The terminal sends the entered text data to the server.

[0368] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[0369] 4. The server sends the response text to the terminal, which displays it to User B.

[0370] This allows the visually impaired to communicate through speech, and the hearing impaired to communicate through text. By utilizing speech recognition, natural language processing, and speech synthesis technologies at each stage, instant and smooth information exchange becomes possible. This invention enables people with and without disabilities to work and study in the same workplace or school, realizing a more inclusive and equal society.

[0371] The processing flow will be explained below.

[0372] Step 1:

[0373] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[0374] Step 2:

[0375] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[0376] Step 3:

[0377] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[0378] Step 4:

[0379] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[0380] Step 5:

[0381] The server sends the converted text data to a natural language processing model to generate an appropriate response text. For example, a natural language processing model can be used to generate a response such as "Yes, I'm fine, thanks to you."

[0382] Step 6:

[0383] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[0384] Step 7:

[0385] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[0386] Step 8:

[0387] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[0388] Step 9:

[0389] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[0390] Step 10:

[0391] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[0392] Step 11:

[0393] The server sends the received text data to a natural language processing model to generate an appropriate response text, such as "Yes, I feel like going for a walk."

[0394] Step 12:

[0395] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[0396] Step 13:

[0397] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[0398] This series of processes realizes a system that allows smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities.

[0399] Example 1

[0400] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0401] The problem that this invention aims to solve is to realize smooth, real-time communication between users with different disabilities, such as the visually or hearing impaired, and between users without disabilities. In particular, the object is to provide a system that integrates voice input, voice recognition, natural language processing, and voice synthesis technologies, allowing users to receive and send information in a format that is suitable for them.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0403] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text and playing it back to the user, thereby enabling smooth communication between the visually impaired and the hearing impaired, and between users with no disabilities.

[0404] "User" refers to a person who uses the system to input voice or text and receives a response.

[0405] "Audio input" refers to audio signals provided by a user to a system through a microphone device.

[0406] "Text data" refers to text information generated by voice recognition technology or text input.

[0407] "Natural language processing model" refers to a machine learning algorithm for generating appropriate response text from text data.

[0408] "Response text" is the response content generated by the natural language processing model expressed as text information.

[0409] "Voice data" refers to a voice signal converted from response text using voice synthesis technology.

[0410] "Speech recognition technology" refers to technology for converting voice data into text data.

[0411] "Speech synthesis technology" refers to technology for converting text data into natural-sounding speech.

[0412] "Microphone device" refers to a device for capturing a user's voice input.

[0413] A "generative AI model" refers to an algorithm that performs natural language processing using advanced machine learning techniques such as large-scale language models.

[0414] "Communication protocol" refers to the rules and procedures used to send and receive data.

[0415] The present invention is a system that enables smooth communication between visually impaired and hearing impaired people, or between users with and without disabilities, by converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data.

[0416] The device is equipped with a microphone device to capture voice input from the user. When the user speaks, the device's microphone converts the voice into digital audio data. The captured audio data is transmitted to the server in real time. This data transmission uses a secure and fast communication protocol (e.g., HTTPS or WebSocket).

[0417] The server converts the voice data into text data using speech recognition technology. Specifically, it uses a speech recognition engine (e.g., a general speech recognition API). The converted text data is sent to a natural language processing model (e.g., a large-scale language model). The natural language processing model generates an appropriate response text from the text data.

[0418] The generated response text is converted back into audio data using speech synthesis technology. A speech synthesis engine (for example, a general speech synthesis API) is used. This audio data is then sent back to the device and played back to the user. This allows visually and hearing impaired people to receive information in a format that is suitable for them.

[0419] Specific examples

[0420] Scenario in which visually impaired user A speaks

[0421] 1. The device captures user A's voice saying "Hello, how are you?" and sends the voice data to the server.

[0422] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[0423] 3. The server sends the converted text data to a natural language processing model and generates the response text "Yes, I'm fine, thanks to you."

[0424] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[0425] 5. The terminal plays the converted voice data to User A.

[0426] Scenario where hearing impaired user B inputs information in text format

[0427] 1. User B enters "It's a nice day today" in text format.

[0428] 2. The terminal sends the entered text data to the server.

[0429] 3. The server sends the input text data to a natural language processing model and generates the response text "Yes, I feel like going for a walk."

[0430] 4. The server sends the generated response text to the terminal, which displays it to User B.

[0431] Prompt Sentence Examples

[0432] Case of visually impaired user A:

[0433] User A says "Hello, how are you?" Generate an appropriate response to this.

[0434] Deaf User B's Case:

[0435] User B has entered the text "It's a nice day today." Generate an appropriate response to this.

[0436] This system is designed to enable smooth communication between visually and hearing impaired people, as well as users with no disabilities, and makes use of voice recognition, natural language processing, and voice synthesis technologies at each stage to enable instant and smooth information exchange.

[0437] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0438] Step 1:

[0439] The device uses a microphone device to capture voice input from the user. When the user says "Hello, how are you?", the device converts the voice into digital voice data. The input is the user's voice, and the output is digital voice data. This voice data is saved in a format such as PCM.

[0440] Step 2:

[0441] The device transmits the captured audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket). The input is the captured digital audio data, and the output is the data sent to the server.

[0442] Step 3:

[0443] The server converts the received voice data into text data using voice recognition technology. Specifically, the conversion is performed using a voice recognition engine. The input is digital voice data, and the output is text data such as "Hello, how are you?" The voice recognition engine analyzes the voice data and converts it into text information.

[0444] Step 4:

[0445] The server sends the converted text data to a natural language processing model. Specifically, it uses a large-scale language model to generate an appropriate response text from the text data. The input is the text data "Hello, how are you?", and the output is the response text "Yes, thanks to you, I'm fine."

[0446] Step 5:

[0447] The server converts the generated response text into voice data. It uses voice synthesis technology to convert text data into voice. Specifically, it uses a voice synthesis engine. The input is the response text "Yes, I'm fine, thanks to you," and the output is digital voice data.

[0448] Step 6:

[0449] The server transmits the converted voice data to the terminal using a secure communication protocol. The input is digital voice data, and the output is data to be transmitted to the terminal.

[0450] Step 7:

[0451] The terminal plays the received audio data, and the user can listen to the audio through the speaker. The input is the digital audio data sent from the server, and the output is the audio played to the user.

[0452] This series of processes enables smooth communication between the visually impaired, the hearing impaired, and users without disabilities. Specifically, the process combines appropriate technologies for each step: voice capture, data transmission, voice recognition, natural language processing, voice synthesis, and voice playback.

[0453] (Application example 1)

[0454] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] Visually and hearing impaired people face challenges in communicating smoothly in virtual stores. In particular, they face difficulties in sharing information with store clerks and other customers in a timely manner, which often leads to stress and inconvenience in the process of obtaining specific information or services. Furthermore, conventional systems do not adequately achieve real-time voice input, text conversion, or appropriate response generation.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0457] In this invention, the server includes means for acquiring voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for supporting communication between customers and shop staff in a virtual store, means for receiving voice input from the user in real time via a smart device and generating an appropriate response, and means for presenting the response generated by the above means to the user. This enables visually impaired and hearing impaired people to communicate smoothly and in real time using voice and text in a virtual store.

[0458] "User" refers to a person or entity that uses the System.

[0459] "Means for obtaining voice input" refers to the technology or device that captures the voice uttered by the user and that the system captures the voice data.

[0460] "Means for converting into text data" refers to a technology or device for converting acquired voice data into text information.

[0461] "Natural language processing model" refers to a machine learning model used to analyze input text data and generate an appropriate response.

[0462] "Response text" refers to text data of a response generated by a natural language processing model.

[0463] "Means for converting into voice data" refers to a technology or device for converting response text into voice data.

[0464] "Means for playing audio data to a user" refers to a technique or device for letting a user hear the generated audio data.

[0465] A "virtual store" refers to a virtual shopping or service space that is developed online.

[0466] "Customer" refers to a user who uses the virtual store.

[0467] "Shop staff" refers to a person or AI avatar set up to interact with customers within a virtual store.

[0468] "Smart device" refers to a device (e.g., smartphone, smart glasses, etc.) that has internet connectivity and can use a wide variety of applications.

[0469] "Means for generating an appropriate response" refers to a technique or device for generating a response to a user's input using a natural language processing model.

[0470] "Means for presenting to the user" refers to a technology or device that visually or audibly displays to the user the response generated through the smart device.

[0471] This invention is a system that supports smooth communication within a virtual store for visually and hearing impaired people. This system allows users to exchange information within the store via voice and text through their own smart devices.

[0472] 1. Program Overview

[0473] The system includes the following elements:

[0474] Acquire voice input: Acquire voice input from the user.

[0475] Speech recognition: Converts voice input into text data.

[0476] Natural Language Processing: Sending text data to a natural language processing model to generate a response.

[0477] Speech synthesis: Converts response text into audio data.

[0478] Audio playback: The generated audio data is played back to the user.

[0479] Text input and response reception: Send text data to the natural language processing model and receive a response.

[0480] Communication support in virtual stores: Supports communication between customers and store staff within virtual stores.

[0481] 2. System configuration

[0482] Voice input acquisition: Voice input is acquired through the microphone of the smart device (smart glasses or smartphone).

[0483] Speech recognition: Uses the Google Speech Recognition API to convert voice data into text data.

[0484] Natural Language Processing: Uses a Transformer model (e.g., rinna / japanese-gpt-1b) to generate appropriate responses to input text.

[0485] Speech synthesis: Use gTTS (Google Text-to-Speech) to convert the generated response text into voice data.

[0486] Audio playback: Use Pygame to play audio data and present it to the user.

[0487] Text input and response reception: The system sends text input to a natural language processing model and receives the generated response, which it displays to the user.

[0488] Communication support in virtual stores: Receives voice input from users in real time via smart devices and generates and presents appropriate responses.

[0489] 3. Specific examples of processing

[0490] As a concrete example, consider a visually impaired person looking for product information in a virtual store:

[0491] 1. A visually impaired person asks the smart glasses, "Tell me about this product."

[0492] 2. The microphone in the smart glasses captures the audio and converts it to text via the Google Speech Recognition API.

[0493] 3. The text data is sent to a natural language processing model (rinna / japanese-gpt-1b), which generates the response, "This product is a state-of-the-art smartphone. Its features include a high-resolution camera and a large-capacity battery."

[0494] 4. The generated response text is converted to speech using gTTS.

[0495] 5. The audio data is played to the user using Pygame, and the user receives information through audio.

[0496] Example prompt sentence:

[0497] "Prompt: User has asked a question about a product. Please provide a brief description of the relevant product information."

[0498] In this way, the system embodying the present invention enables visually impaired and hearing impaired people to smoothly obtain information and communicate within a virtual store.

[0499] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0500] Step 1:

[0501] The device captures voice input from the user using a microphone. The input is voice data and prepares it to be sent to a speech recognition API. Specifically, the microphone captures the user's voice, converts the data into a digital format, and stores it in memory.

[0502] Step 2:

[0503] The device sends the captured voice data to the Google Speech Recognition API. The input is digital voice data, and the output is corresponding text data. Specifically, the digital voice data is sent to the API endpoint, which performs real-time speech-to-text conversion.

[0504] Step 3:

[0505] The server receives text data from the Google Speech Recognition API and sends it to the natural language processing model. The input is the converted text data, and the output is the generated response text. Specifically, the text data is sent to the API of the natural language processing model, which generates appropriate text as a response.

[0506] Step 4:

[0507] The server sends the generated response text to the Google Text-to-Speech (gTTS) API and converts it into voice data. The input is the response text and the output is voice data. Specifically, the response text is sent to the gTTS API, which uses speech synthesis technology to convert the text into voice.

[0508] Step 5:

[0509] The device plays the generated audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to play the audio data, and the audio is output through the device's speaker.

[0510] Step 6:

[0511] The user inputs text data arbitrarily. The input is text data, which is then prepared for transmission to the natural language processing model. Specifically, the text data is input using the device's keyboard or voice recognition function.

[0512] Step 7:

[0513] The server sends the input text data to the natural language processing model and receives a response. The input is text data and the output is response text. Specifically, the server sends text data to the natural language processing model and receives appropriate text as a response.

[0514] Step 8:

[0515] The server converts the received response text into voice data. The input is the text data to be converted, and the output is voice data. Specifically, the server uses the gTTS API to convert text data into voice data.

[0516] Step 9:

[0517] The device plays the converted audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to output the audio through the device's speakers.

[0518] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0519] This invention is a communication support system that combines an emotion engine, which converts voice input into text data, generates an appropriate response using a natural language processing model, and converts that response into voice data, thereby realizing smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities. Furthermore, by using the emotion engine, emotions can be added to the user's utterances, further supporting natural communication.

[0520] In this system, the terminal is equipped with a microphone for capturing voice input from the user and acquires the voice input. The captured voice data is sent to a server in real time. The server uses voice recognition technology to convert the voice input into text data. The converted text data is then sent to an emotion engine, which recognizes the user's emotion. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), which generates a response text. The generated response text is again processed within the server and converted into voice data using voice synthesis technology. The terminal plays the converted voice data to the user. In this way, visually and hearing impaired people can receive information in a format that is suitable for them.

[0521] As a specific example, suppose visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data, "Hello, how are you?" The emotion engine then recognizes user A's emotion from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text, "Yes, I'm fine, thanks to you." At this time, a response is generated that includes the "friendly" emotion. The server then converts the response text into voice data and sends the voice data to the device. The device plays the voice data to user A.

[0522] Meanwhile, consider the case where hearing-impaired User B responds by text input. User B inputs "It's a nice day today, isn't it?" in text format. The device sends the input text data to the server. The server sends the text data to the emotion engine, which recognizes the emotion "joy" from User B's input. The server sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk." The server then sends the response text back to the device in text format. The device displays the received text data to User B.

[0523] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.The present invention aims to improve the quality of communication, enable people with disabilities to share information equally, and reduce social isolation.

[0524] The processing flow will be explained below.

[0525] Step 1:

[0526] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[0527] Step 2:

[0528] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[0529] Step 3:

[0530] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[0531] Step 4:

[0532] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[0533] Step 5:

[0534] The server sends the converted text data to an emotion engine to recognize the user's emotion, for example, using the emotion engine to recognize the "friendly" emotion.

[0535] Step 6:

[0536] The server sends the text data and the recognized emotion data to a natural language processing model to generate an appropriate response text. For example, the natural language processing model generates a response such as "Yes, thanks to you, I'm fine."

[0537] Step 7:

[0538] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[0539] Step 8:

[0540] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[0541] Step 9:

[0542] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[0543] Step 10:

[0544] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[0545] Step 11:

[0546] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[0547] Step 12:

[0548] The server sends the received text data to the emotion engine and recognizes the emotion of User B. For example, it recognizes the emotion of "joy."

[0549] Step 13:

[0550] The server sends the text data and recognized emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[0551] Step 14:

[0552] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[0553] Step 15:

[0554] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[0555] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.

[0556] Example 2

[0557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] The present invention aims to enable smooth communication between users who have communication difficulties, such as the visually or hearing impaired. In particular, by combining technologies of speech recognition, emotion recognition, natural language processing, and speech synthesis, the system generates natural responses by adding emotion to user utterances, supporting two-way communication. Another objective is to provide a system that allows information to be transmitted seamlessly even when users use different input methods (voice or text).

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0560] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to an emotion recognition engine and acquiring emotion data, means for transmitting the emotion data and the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text generated by the seventh means, converting it into voice data, and playing it back to the user. This enables visually impaired and hearing impaired people to communicate naturally with emotions.

[0561] "Voice input" is the act or means of obtaining a user's spoken words as a digital signal.

[0562] "Text data" is a data format in which voice input is converted into text information.

[0563] An "emotion recognition engine" is software or a system for analyzing and identifying user emotions from text data.

[0564] A "natural language processing model" is a machine learning model that analyzes given text data and generates responses based on grammar and context.

[0565] "Response text" is text data generated in response to a user input.

[0566] "Audio data" is a data format in which text data is converted into a digital audio signal.

[0567] "Voice recognition technology" is a technology for converting voice input into text data.

[0568] "Speech synthesis technology" is a technology for converting text data into voice data.

[0569] "User" refers to an individual or entity that uses this system, and includes the visually impaired and the hearing impaired.

[0570] This invention is a communication support system that incorporates an emotion recognition engine. By converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data, it enables smooth communication between the visually impaired, the hearing impaired, and people with and without disabilities. Furthermore, by using an emotion recognition engine, it is possible to assign emotions to the user's speech, supporting more natural communication.

[0571] This system consists of the following main hardware and software components: First, the device is equipped with a microphone that captures voice input from the user. This voice data is sent to a server in real time. The server converts the voice data into text data using voice recognition technology (e.g., a voice recognition API). Next, the converted text data is sent to an emotion recognition engine (e.g., an emotion recognition API), which analyzes the user's emotions. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), and a response text is generated. The generated response text is again processed within the server and converted into voice data using voice synthesis technology (e.g., a voice synthesis API). The device then plays the converted voice data to the user.

[0572] As a specific example, consider the case where visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?" The emotion recognition engine then recognizes user A's emotions from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you." The "friendly" emotion is also taken into consideration at this time. The server then converts the response text into voice data and sends the voice data to the device. The device then plays the voice data to user A.

[0573] On the other hand, consider the case where hearing-impaired user B inputs "It's a nice day today." User B inputs in text format, and the device sends the text data to the server. The server sends the text data to an emotion recognition engine, which recognizes the emotion "joy" from user B's input. The server sends the text data and emotion data to a natural language processing model, which generates a response text: "Yes, I feel like going for a walk." The server then sends the generated response text back to the device in text format, and the device displays the received text data to user B.

[0574] Example prompt sentence:

[0575] 1. "When visually impaired person A says, 'Hello, how are you?' Please generate a response text. Please make it a little friendly."

[0576] 2. "Generate a response to the user's input 'What a nice day today!' with the emotion 'Joy'."

[0577] This series of processes enables all users, including those with visual or hearing impairments, to communicate naturally and with emotion.

[0578] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0579] Step 1:

[0580] The user speaks into the microphone, for example, "Hello, how are you?" This is the voice input to the system.

[0581] Input: Speech data ("Hello, how are you?")

[0582] Output: Captured audio data

[0583] Step 2:

[0584] The device uses a microphone to capture the user's voice and obtains the voice data in real time.

[0585] Input: Speech data ("Hello, how are you?")

[0586] Output: Digital audio data

[0587] Step 3:

[0588] The device then transmits the captured audio data over the internet to a server, which is often encrypted to ensure secure transmission.

[0589] Input: Digital audio data

[0590] Output: Audio data sent to the server

[0591] Step 4:

[0592] The server converts the voice data into text data using voice recognition technology (for example, a voice recognition API), which recognizes the voice as text.

[0593] Input: Audio data

[0594] Output: Text data ("Hello, how are you?")

[0595] Step 5:

[0596] The server sends the text data to an emotion recognition engine (e.g., an emotion recognition API) to analyze the user's emotion.

[0597] Input: Text data ("Hello, how are you?")

[0598] Output: Emotion data ("Friendly")

[0599] Step 6:

[0600] The server sends the text data and emotion data to a natural language processing model (e.g., a large-scale language model) to generate a response text.

[0601] Input: Text data ("Hello, how are you?"), Emotion data ("Friendly")

[0602] Output: Response text ("Yes, I'm fine, thanks to you.")

[0603] Step 7:

[0604] The server converts the generated response text into voice data using a voice synthesis technology (for example, a voice synthesis API).

[0605] Input: Response text ("Yes, I'm fine, thanks to you.")

[0606] Output: Audio data

[0607] Step 8:

[0608] The server then sends the audio data back to the device over the internet, where it is again transferred securely.

[0609] Input: Audio data

[0610] Output: Audio data transferred to the device

[0611] Step 9:

[0612] The terminal plays the received audio data over a speaker to provide feedback to the user.

[0613] Input: Audio data

[0614] Output: Played speech ("Yes, I'm fine, thanks to you.")

[0615] Through the above steps, all users, including those with visual or hearing impairments, can communicate naturally and with emotion.

[0616] (Application example 2)

[0617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] Visually and hearing impaired people face challenges when communicating with others in public places or stores, such as difficulty sending and receiving information smoothly. They also face difficulties in communicating naturally and emotionally, which can lead to a sense of social isolation. Therefore, there is a need for a new system that allows people with disabilities to share information equally and realize natural and smooth dialogue.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0620] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for recognizing emotions from the text data using an emotion engine, means for generating the response text to which the emotions have been added, and means for converting the response text into voice data, thereby enabling visually impaired and hearing impaired people to communicate naturally and smoothly in public places such as brick-and-mortar stores.

[0621] "User" refers to a person who uses the system.

[0622] "Voice input" refers to voice data collected from a user's speech through an input device such as a microphone.

[0623] "Voice recognition technology" refers to technology that analyzes voice data and converts its content into text data.

[0624] "Text data" refers to data that has been converted from voice input into a string of characters using voice recognition technology.

[0625] A "natural language processing model" refers to an artificial intelligence model that analyzes text data, understands its meaning, and generates an appropriate response.

[0626] "Response Text" refers to the response string generated by a natural language processing model.

[0627] An "emotion engine" refers to technology that recognizes a user's emotions from text data and analyzes those emotions.

[0628] "Audio data" refers to data that has been converted back into audio format from text data using speech synthesis technology.

[0629] "Speech synthesis technology" refers to technology for outputting text data as voice data.

[0630] "Playing back to the user" means letting the user hear the synthesized voice data via a speaker, earphones, or the like.

[0631] "System" refers to a collection of devices and software that performs a series of processes including voice input acquisition, speech recognition, natural language processing, emotion recognition, and speech synthesis.

[0632] As an embodiment of the present invention, we will explain a system that enables visually impaired and hearing impaired people to communicate smoothly in physical stores. This system uses smart glasses and related terminals owned by users, and a server.

[0633] First, the user inputs voice through the smart glasses. The smart glasses have a built-in microphone that captures the user's voice. The captured voice data is sent to the server in real time. The server then converts the voice data into text data using the speech_recognition library.

[0634] Next, this text data is sent to a natural language processing model, which is a generative AI model. The server also contains an emotion engine that analyzes the user's emotions from the received text data. The emotion engine recognizes emotions such as "friendly" or "joyful" and adds this emotional information to the natural language processing model.

[0635] A natural language processing model takes emotion into account to generate an appropriate response text. This response is then converted into audio data on the server and sent to the smart glasses, which then play the response aloud to the user using a bone conduction speaker.

[0636] As a concrete example, consider the case where a user wears smart glasses and says, "Excuse me, where can I find the bread?" while in a store. The smart glasses capture the voice and send it to the server. The server performs speech recognition and converts it into text data, generating the text data "Excuse me, where can I find the bread." This text data is then sent to a natural language processing model by an emotion engine along with a "friendly" emotion. The response generated is "It's near the refrigerated section. Can I help you? You're very welcome." Finally, the smart glasses play this response to the user through a bone conduction speaker.

[0637] Example prompt sentence:

[0638] The user says, "Excuse me, where is the bread?" Please respond with a friendly sentiment.

[0639] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0640] Step 1:

[0641] The user inputs voice through the smart glasses. The microphone built into the smart glasses captures the user's voice. The input is the user's voice data, and the output is the captured voice data. A specific example of this is when the user says, "Excuse me, where is the bread?"

[0642] Step 2:

[0643] The captured audio data is sent to the server in real time. The input is the audio data captured by the smart glasses, and the output is the audio data sent to the server. Specifically, the smart glasses transfer the audio data to the server via Wi-Fi or Bluetooth.

[0644] Step 3:

[0645] The server converts the voice data into text data using the speech_recognition library. The input is the voice data sent to the server, and the output is the text data "Excuse me, where is the bread?". Specifically, the server receives the voice data and executes the process of converting it into text.

[0646] Step 4:

[0647] The server sends this text data to a natural language processing model, which is a generative AI model. The input is the converted text data, and the output is the transmission of the text data to the natural language processing model. Specifically, the server uses an API to send the text data to the model.

[0648] Step 5:

[0649] The emotion engine of the server recognizes the user's emotion from the text data. The input is the text data sent to the natural language processing model, and the output is the emotion recognition data "friendly." In specific operations, the emotion engine of the server analyzes the text data and identifies the emotion.

[0650] Step 6:

[0651] The natural language processing model takes emotion into account and generates a response text. The input is the emotion recognition data "friendly" and the text data "Excuse me, where is the bread?", and the output is the response text "It's near the refrigerated section. Can I help you? You're most welcome." In concrete terms, the natural language processing model takes emotion into account and generates an appropriate response.

[0652] Step 7:

[0653] The server converts the generated response text into voice data. The input is the response text and the output is voice data. Specifically, the server uses voice synthesis technology to convert the text data into voice data.

[0654] Step 8:

[0655] The converted voice data is sent to the smart glasses. The input is the voice data generated by the server, and the output is the voice data sent to the smart glasses. Specifically, the server transfers the voice data to the smart glasses again via Wi-Fi or Bluetooth.

[0656] Step 9:

[0657] The smart glasses use a bone conduction speaker to play audio data to the user. The input is the audio data sent to the smart glasses, and the output is the audio heard by the user. Specifically, the bone conduction speaker of the smart glasses plays the audio to the user, saying, "You're near the refrigerated section. Can you help me? You're very welcome."

[0658] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0659] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0660] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0661] [Third embodiment]

[0662] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0663] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0664] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0665] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0666] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0667] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0668] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0669] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0670] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0671] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0672] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0673] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0674] The system of the present invention converts voice input into text data, generates an appropriate response using a natural language processing model, and then converts that response back into voice data, thereby enabling smooth communication between visually impaired and hearing impaired people, or between people with and without disabilities.

[0675] In this system, the terminal is equipped with a microphone to capture voice input from the user, and the captured voice data is transmitted to the server in real time. The server then uses voice recognition technology to convert the voice input into text data.

[0676] The server then sends the converted text data to a natural language processing model (e.g., a large-scale language model) to generate an appropriate response text, which is then processed again within the server and converted into voice data using speech synthesis technology.

[0677] The terminal then plays the converted audio data back to the user, allowing visually and hearing impaired people to receive information in a format that is convenient for them.

[0678] As a specific example, consider a scenario in which visually impaired user A says, "Hello, how are you?"

[0679] 1. The device captures user A's voice with a microphone and sends the voice data to the server.

[0680] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[0681] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you."

[0682] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[0683] 5. The terminal plays the audio data to User A.

[0684] On the other hand, consider the case where hearing-impaired user B inputs and receives information in text format.

[0685] 1. User B enters "It's a nice day today" in text format.

[0686] 2. The terminal sends the entered text data to the server.

[0687] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[0688] 4. The server sends the response text to the terminal, which displays it to User B.

[0689] This allows the visually impaired to communicate through speech, and the hearing impaired to communicate through text. By utilizing speech recognition, natural language processing, and speech synthesis technologies at each stage, instant and smooth information exchange becomes possible. This invention enables people with and without disabilities to work and study in the same workplace or school, realizing a more inclusive and equal society.

[0690] The processing flow will be explained below.

[0691] Step 1:

[0692] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[0693] Step 2:

[0694] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[0695] Step 3:

[0696] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[0697] Step 4:

[0698] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[0699] Step 5:

[0700] The server sends the converted text data to a natural language processing model to generate an appropriate response text. For example, a natural language processing model can be used to generate a response such as "Yes, I'm fine, thanks to you."

[0701] Step 6:

[0702] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[0703] Step 7:

[0704] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[0705] Step 8:

[0706] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[0707] Step 9:

[0708] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[0709] Step 10:

[0710] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[0711] Step 11:

[0712] The server sends the received text data to a natural language processing model to generate an appropriate response text, such as "Yes, I feel like going for a walk."

[0713] Step 12:

[0714] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[0715] Step 13:

[0716] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[0717] This series of processes realizes a system that allows smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities.

[0718] Example 1

[0719] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0720] The problem that this invention aims to solve is to realize smooth, real-time communication between users with different disabilities, such as the visually or hearing impaired, and between users without disabilities. In particular, the object is to provide a system that integrates voice input, voice recognition, natural language processing, and voice synthesis technologies, allowing users to receive and send information in a format that is suitable for them.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0722] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text and playing it back to the user, thereby enabling smooth communication between the visually impaired and the hearing impaired, and between users with no disabilities.

[0723] "User" refers to a person who uses the system to input voice or text and receives a response.

[0724] "Audio input" refers to audio signals provided by a user to a system through a microphone device.

[0725] "Text data" refers to text information generated by voice recognition technology or text input.

[0726] "Natural language processing model" refers to a machine learning algorithm for generating appropriate response text from text data.

[0727] "Response text" is the response content generated by the natural language processing model expressed as text information.

[0728] "Voice data" refers to a voice signal converted from response text using voice synthesis technology.

[0729] "Speech recognition technology" refers to technology for converting voice data into text data.

[0730] "Speech synthesis technology" refers to technology for converting text data into natural-sounding speech.

[0731] "Microphone device" refers to a device for capturing a user's voice input.

[0732] A "generative AI model" refers to an algorithm that performs natural language processing using advanced machine learning techniques such as large-scale language models.

[0733] "Communication protocol" refers to the rules and procedures used to send and receive data.

[0734] The present invention is a system that enables smooth communication between visually impaired and hearing impaired people, or between users with and without disabilities, by converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data.

[0735] The device is equipped with a microphone device to capture voice input from the user. When the user speaks, the device's microphone converts the voice into digital audio data. The captured audio data is transmitted to the server in real time. This data transmission uses a secure and fast communication protocol (e.g., HTTPS or WebSocket).

[0736] The server converts the voice data into text data using speech recognition technology. Specifically, it uses a speech recognition engine (e.g., a general speech recognition API). The converted text data is sent to a natural language processing model (e.g., a large-scale language model). The natural language processing model generates an appropriate response text from the text data.

[0737] The generated response text is converted back into audio data using speech synthesis technology. A speech synthesis engine (for example, a general speech synthesis API) is used. This audio data is then sent back to the device and played back to the user. This allows visually and hearing impaired people to receive information in a format that is suitable for them.

[0738] Specific examples

[0739] Scenario in which visually impaired user A speaks

[0740] 1. The device captures user A's voice saying "Hello, how are you?" and sends the voice data to the server.

[0741] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[0742] 3. The server sends the converted text data to a natural language processing model and generates the response text "Yes, I'm fine, thanks to you."

[0743] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[0744] 5. The terminal plays the converted voice data to User A.

[0745] Scenario where hearing impaired user B inputs information in text format

[0746] 1. User B enters "It's a nice day today" in text format.

[0747] 2. The terminal sends the entered text data to the server.

[0748] 3. The server sends the input text data to a natural language processing model and generates the response text "Yes, I feel like going for a walk."

[0749] 4. The server sends the generated response text to the terminal, which displays it to User B.

[0750] Prompt Sentence Examples

[0751] Case of visually impaired user A:

[0752] User A says "Hello, how are you?" Generate an appropriate response to this.

[0753] Deaf User B's Case:

[0754] User B has entered the text "It's a nice day today." Generate an appropriate response to this.

[0755] This system is designed to enable smooth communication between visually and hearing impaired people, as well as users with no disabilities, and makes use of voice recognition, natural language processing, and voice synthesis technologies at each stage to enable instant and smooth information exchange.

[0756] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0757] Step 1:

[0758] The device uses a microphone device to capture voice input from the user. When the user says "Hello, how are you?", the device converts the voice into digital voice data. The input is the user's voice, and the output is digital voice data. This voice data is saved in a format such as PCM.

[0759] Step 2:

[0760] The device transmits the captured audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket). The input is the captured digital audio data, and the output is the data sent to the server.

[0761] Step 3:

[0762] The server converts the received voice data into text data using voice recognition technology. Specifically, the conversion is performed using a voice recognition engine. The input is digital voice data, and the output is text data such as "Hello, how are you?" The voice recognition engine analyzes the voice data and converts it into text information.

[0763] Step 4:

[0764] The server sends the converted text data to a natural language processing model. Specifically, it uses a large-scale language model to generate an appropriate response text from the text data. The input is the text data "Hello, how are you?", and the output is the response text "Yes, thanks to you, I'm fine."

[0765] Step 5:

[0766] The server converts the generated response text into voice data. It uses voice synthesis technology to convert text data into voice. Specifically, it uses a voice synthesis engine. The input is the response text "Yes, I'm fine, thanks to you," and the output is digital voice data.

[0767] Step 6:

[0768] The server transmits the converted voice data to the terminal using a secure communication protocol. The input is digital voice data, and the output is data to be transmitted to the terminal.

[0769] Step 7:

[0770] The terminal plays the received audio data, and the user can listen to the audio through the speaker. The input is the digital audio data sent from the server, and the output is the audio played to the user.

[0771] This series of processes enables smooth communication between the visually impaired, the hearing impaired, and users without disabilities. Specifically, the process combines appropriate technologies for each step: voice capture, data transmission, voice recognition, natural language processing, voice synthesis, and voice playback.

[0772] (Application example 1)

[0773] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0774] Visually and hearing impaired people face challenges in communicating smoothly in virtual stores. In particular, they face difficulties in sharing information with store clerks and other customers in a timely manner, which often leads to stress and inconvenience in the process of obtaining specific information or services. Furthermore, conventional systems do not adequately achieve real-time voice input, text conversion, or appropriate response generation.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0776] In this invention, the server includes means for acquiring voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for supporting communication between customers and shop staff in a virtual store, means for receiving voice input from the user in real time via a smart device and generating an appropriate response, and means for presenting the response generated by the above means to the user. This enables visually impaired and hearing impaired people to communicate smoothly and in real time using voice and text in a virtual store.

[0777] "User" refers to a person or entity that uses the System.

[0778] "Means for obtaining voice input" refers to the technology or device that captures the voice uttered by the user and that the system captures the voice data.

[0779] "Means for converting into text data" refers to a technology or device for converting acquired voice data into text information.

[0780] "Natural language processing model" refers to a machine learning model used to analyze input text data and generate an appropriate response.

[0781] "Response text" refers to text data of a response generated by a natural language processing model.

[0782] "Means for converting into voice data" refers to a technology or device for converting response text into voice data.

[0783] "Means for playing audio data to a user" refers to a technique or device for letting a user hear the generated audio data.

[0784] A "virtual store" refers to a virtual shopping or service space that is developed online.

[0785] "Customer" refers to a user who uses the virtual store.

[0786] "Shop staff" refers to a person or AI avatar set up to interact with customers within a virtual store.

[0787] "Smart device" refers to a device (e.g., smartphone, smart glasses, etc.) that has internet connectivity and can use a wide variety of applications.

[0788] "Means for generating an appropriate response" refers to a technique or device for generating a response to a user's input using a natural language processing model.

[0789] "Means for presenting to the user" refers to a technology or device that visually or audibly displays to the user the response generated through the smart device.

[0790] This invention is a system that supports smooth communication within a virtual store for visually and hearing impaired people. This system allows users to exchange information within the store via voice and text through their own smart devices.

[0791] 1. Program Overview

[0792] The system includes the following elements:

[0793] Acquire voice input: Acquire voice input from the user.

[0794] Speech recognition: Converts voice input into text data.

[0795] Natural Language Processing: Sending text data to a natural language processing model to generate a response.

[0796] Speech synthesis: Converts response text into audio data.

[0797] Audio playback: The generated audio data is played back to the user.

[0798] Text input and response reception: Send text data to the natural language processing model and receive a response.

[0799] Communication support in virtual stores: Supports communication between customers and store staff within virtual stores.

[0800] 2. System configuration

[0801] Voice input acquisition: Voice input is acquired through the microphone of the smart device (smart glasses or smartphone).

[0802] Speech recognition: Uses the Google Speech Recognition API to convert voice data into text data.

[0803] Natural Language Processing: Uses a Transformer model (e.g., rinna / japanese-gpt-1b) to generate appropriate responses to input text.

[0804] Speech synthesis: Use gTTS (Google Text-to-Speech) to convert the generated response text into voice data.

[0805] Audio playback: Use Pygame to play audio data and present it to the user.

[0806] Text input and response reception: The system sends text input to a natural language processing model and receives the generated response, which it displays to the user.

[0807] Communication support in virtual stores: Receives voice input from users in real time via smart devices and generates and presents appropriate responses.

[0808] 3. Specific examples of processing

[0809] As a concrete example, consider a visually impaired person looking for product information in a virtual store:

[0810] 1. A visually impaired person asks the smart glasses, "Tell me about this product."

[0811] 2. The microphone in the smart glasses captures the audio and converts it to text via the Google Speech Recognition API.

[0812] 3. The text data is sent to a natural language processing model (rinna / japanese-gpt-1b), which generates the response, "This product is a state-of-the-art smartphone. Its features include a high-resolution camera and a large-capacity battery."

[0813] 4. The generated response text is converted to speech using gTTS.

[0814] 5. The audio data is played to the user using Pygame, and the user receives information through audio.

[0815] Example prompt sentence:

[0816] "Prompt: User has asked a question about a product. Please provide a brief description of the relevant product information."

[0817] In this way, the system embodying the present invention enables visually impaired and hearing impaired people to smoothly obtain information and communicate within a virtual store.

[0818] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0819] Step 1:

[0820] The device captures voice input from the user using a microphone. The input is voice data and prepares it to be sent to a speech recognition API. Specifically, the microphone captures the user's voice, converts the data into a digital format, and stores it in memory.

[0821] Step 2:

[0822] The device sends the captured voice data to the Google Speech Recognition API. The input is digital voice data, and the output is corresponding text data. Specifically, the digital voice data is sent to the API endpoint, which performs real-time speech-to-text conversion.

[0823] Step 3:

[0824] The server receives text data from the Google Speech Recognition API and sends it to the natural language processing model. The input is the converted text data, and the output is the generated response text. Specifically, the text data is sent to the API of the natural language processing model, which generates appropriate text as a response.

[0825] Step 4:

[0826] The server sends the generated response text to the Google Text-to-Speech (gTTS) API and converts it into voice data. The input is the response text and the output is voice data. Specifically, the response text is sent to the gTTS API, which uses speech synthesis technology to convert the text into voice.

[0827] Step 5:

[0828] The device plays the generated audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to play the audio data, and the audio is output through the device's speaker.

[0829] Step 6:

[0830] The user inputs text data arbitrarily. The input is text data, which is then prepared for transmission to the natural language processing model. Specifically, the text data is input using the device's keyboard or voice recognition function.

[0831] Step 7:

[0832] The server sends the input text data to the natural language processing model and receives a response. The input is text data and the output is response text. Specifically, the server sends text data to the natural language processing model and receives appropriate text as a response.

[0833] Step 8:

[0834] The server converts the received response text into voice data. The input is the text data to be converted, and the output is voice data. Specifically, the server uses the gTTS API to convert text data into voice data.

[0835] Step 9:

[0836] The device plays the converted audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to output the audio through the device's speakers.

[0837] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0838] This invention is a communication support system that combines an emotion engine, which converts voice input into text data, generates an appropriate response using a natural language processing model, and converts that response into voice data, thereby realizing smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities. Furthermore, by using the emotion engine, emotions can be added to the user's utterances, further supporting natural communication.

[0839] In this system, the terminal is equipped with a microphone for capturing voice input from the user and acquires the voice input. The captured voice data is sent to a server in real time. The server uses voice recognition technology to convert the voice input into text data. The converted text data is then sent to an emotion engine, which recognizes the user's emotion. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), which generates a response text. The generated response text is again processed within the server and converted into voice data using voice synthesis technology. The terminal plays the converted voice data to the user. In this way, visually and hearing impaired people can receive information in a format that is suitable for them.

[0840] As a specific example, suppose visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data, "Hello, how are you?" The emotion engine then recognizes user A's emotion from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text, "Yes, I'm fine, thanks to you." At this time, a response is generated that includes the "friendly" emotion. The server then converts the response text into voice data and sends the voice data to the device. The device plays the voice data to user A.

[0841] Meanwhile, consider the case where hearing-impaired User B responds by text input. User B inputs "It's a nice day today, isn't it?" in text format. The device sends the input text data to the server. The server sends the text data to the emotion engine, which recognizes the emotion "joy" from User B's input. The server sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk." The server then sends the response text back to the device in text format. The device displays the received text data to User B.

[0842] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.The present invention aims to improve the quality of communication, enable people with disabilities to share information equally, and reduce social isolation.

[0843] The processing flow will be explained below.

[0844] Step 1:

[0845] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[0846] Step 2:

[0847] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[0848] Step 3:

[0849] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[0850] Step 4:

[0851] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[0852] Step 5:

[0853] The server sends the converted text data to an emotion engine to recognize the user's emotion, for example, using the emotion engine to recognize the "friendly" emotion.

[0854] Step 6:

[0855] The server sends the text data and the recognized emotion data to a natural language processing model to generate an appropriate response text. For example, the natural language processing model generates a response such as "Yes, thanks to you, I'm fine."

[0856] Step 7:

[0857] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[0858] Step 8:

[0859] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[0860] Step 9:

[0861] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[0862] Step 10:

[0863] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[0864] Step 11:

[0865] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[0866] Step 12:

[0867] The server sends the received text data to the emotion engine and recognizes the emotion of User B. For example, it recognizes the emotion of "joy."

[0868] Step 13:

[0869] The server sends the text data and recognized emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[0870] Step 14:

[0871] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[0872] Step 15:

[0873] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[0874] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.

[0875] Example 2

[0876] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0877] The present invention aims to enable smooth communication between users who have communication difficulties, such as the visually or hearing impaired. In particular, by combining technologies of speech recognition, emotion recognition, natural language processing, and speech synthesis, the system generates natural responses by adding emotion to user utterances, supporting two-way communication. Another objective is to provide a system that allows information to be transmitted seamlessly even when users use different input methods (voice or text).

[0878] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0879] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to an emotion recognition engine and acquiring emotion data, means for transmitting the emotion data and the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text generated by the seventh means, converting it into voice data, and playing it back to the user. This enables visually impaired and hearing impaired people to communicate naturally with emotions.

[0880] "Voice input" is the act or means of obtaining a user's spoken words as a digital signal.

[0881] "Text data" is a data format in which voice input is converted into text information.

[0882] An "emotion recognition engine" is software or a system for analyzing and identifying user emotions from text data.

[0883] A "natural language processing model" is a machine learning model that analyzes given text data and generates responses based on grammar and context.

[0884] "Response text" is text data generated in response to a user input.

[0885] "Audio data" is a data format in which text data is converted into a digital audio signal.

[0886] "Voice recognition technology" is a technology for converting voice input into text data.

[0887] "Speech synthesis technology" is a technology for converting text data into voice data.

[0888] "User" refers to an individual or entity that uses this system, and includes the visually impaired and the hearing impaired.

[0889] This invention is a communication support system that incorporates an emotion recognition engine. By converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data, it enables smooth communication between the visually impaired, the hearing impaired, and people with and without disabilities. Furthermore, by using an emotion recognition engine, it is possible to assign emotions to the user's speech, supporting more natural communication.

[0890] This system consists of the following main hardware and software components: First, the device is equipped with a microphone that captures voice input from the user. This voice data is sent to a server in real time. The server converts the voice data into text data using voice recognition technology (e.g., a voice recognition API). Next, the converted text data is sent to an emotion recognition engine (e.g., an emotion recognition API), which analyzes the user's emotions. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), and a response text is generated. The generated response text is again processed within the server and converted into voice data using voice synthesis technology (e.g., a voice synthesis API). The device then plays the converted voice data to the user.

[0891] As a specific example, consider the case where visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?" The emotion recognition engine then recognizes user A's emotions from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you." The "friendly" emotion is also taken into consideration at this time. The server then converts the response text into voice data and sends the voice data to the device. The device then plays the voice data to user A.

[0892] On the other hand, consider the case where hearing-impaired user B inputs "It's a nice day today." User B inputs in text format, and the device sends the text data to the server. The server sends the text data to an emotion recognition engine, which recognizes the emotion "joy" from user B's input. The server sends the text data and emotion data to a natural language processing model, which generates a response text: "Yes, I feel like going for a walk." The server then sends the generated response text back to the device in text format, and the device displays the received text data to user B.

[0893] Example prompt sentence:

[0894] 1. "When visually impaired person A says, 'Hello, how are you?' Please generate a response text. Please make it a little friendly."

[0895] 2. "Generate a response to the user's input 'What a nice day today!' with the emotion 'Joy'."

[0896] This series of processes enables all users, including those with visual or hearing impairments, to communicate naturally and with emotion.

[0897] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0898] Step 1:

[0899] The user speaks into the microphone, for example, "Hello, how are you?" This is the voice input to the system.

[0900] Input: Speech data ("Hello, how are you?")

[0901] Output: Captured audio data

[0902] Step 2:

[0903] The device uses a microphone to capture the user's voice and obtains the voice data in real time.

[0904] Input: Speech data ("Hello, how are you?")

[0905] Output: Digital audio data

[0906] Step 3:

[0907] The device then transmits the captured audio data over the internet to a server, which is often encrypted to ensure secure transmission.

[0908] Input: Digital audio data

[0909] Output: Audio data sent to the server

[0910] Step 4:

[0911] The server converts the voice data into text data using voice recognition technology (for example, a voice recognition API), which recognizes the voice as text.

[0912] Input: Audio data

[0913] Output: Text data ("Hello, how are you?")

[0914] Step 5:

[0915] The server sends the text data to an emotion recognition engine (e.g., an emotion recognition API) to analyze the user's emotion.

[0916] Input: Text data ("Hello, how are you?")

[0917] Output: Emotion data ("Friendly")

[0918] Step 6:

[0919] The server sends the text data and emotion data to a natural language processing model (e.g., a large-scale language model) to generate a response text.

[0920] Input: Text data ("Hello, how are you?"), Emotion data ("Friendly")

[0921] Output: Response text ("Yes, I'm fine, thanks to you.")

[0922] Step 7:

[0923] The server converts the generated response text into voice data using a voice synthesis technology (for example, a voice synthesis API).

[0924] Input: Response text ("Yes, I'm fine, thanks to you.")

[0925] Output: Audio data

[0926] Step 8:

[0927] The server then sends the audio data back to the device over the internet, where it is again transferred securely.

[0928] Input: Audio data

[0929] Output: Audio data transferred to the device

[0930] Step 9:

[0931] The terminal plays the received audio data over a speaker to provide feedback to the user.

[0932] Input: Audio data

[0933] Output: Played speech ("Yes, I'm fine, thanks to you.")

[0934] Through the above steps, all users, including those with visual or hearing impairments, can communicate naturally and with emotion.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] Visually and hearing impaired people face challenges when communicating with others in public places or stores, such as difficulty sending and receiving information smoothly. They also face difficulties in communicating naturally and emotionally, which can lead to a sense of social isolation. Therefore, there is a need for a new system that allows people with disabilities to share information equally and realize natural and smooth dialogue.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0939] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for recognizing emotions from the text data using an emotion engine, means for generating the response text to which the emotions have been added, and means for converting the response text into voice data, thereby enabling visually impaired and hearing impaired people to communicate naturally and smoothly in public places such as brick-and-mortar stores.

[0940] "User" refers to a person who uses the system.

[0941] "Voice input" refers to voice data collected from a user's speech through an input device such as a microphone.

[0942] "Voice recognition technology" refers to technology that analyzes voice data and converts its content into text data.

[0943] "Text data" refers to data that has been converted from voice input into a string of characters using voice recognition technology.

[0944] A "natural language processing model" refers to an artificial intelligence model that analyzes text data, understands its meaning, and generates an appropriate response.

[0945] "Response Text" refers to the response string generated by a natural language processing model.

[0946] An "emotion engine" refers to technology that recognizes a user's emotions from text data and analyzes those emotions.

[0947] "Audio data" refers to data that has been converted back into audio format from text data using speech synthesis technology.

[0948] "Speech synthesis technology" refers to technology for outputting text data as voice data.

[0949] "Playing back to the user" means letting the user hear the synthesized voice data via a speaker, earphones, or the like.

[0950] "System" refers to a collection of devices and software that performs a series of processes including voice input acquisition, speech recognition, natural language processing, emotion recognition, and speech synthesis.

[0951] As an embodiment of the present invention, we will explain a system that enables visually impaired and hearing impaired people to communicate smoothly in physical stores. This system uses smart glasses and related terminals owned by users, and a server.

[0952] First, the user inputs voice through the smart glasses. The smart glasses have a built-in microphone that captures the user's voice. The captured voice data is sent to the server in real time. The server then converts the voice data into text data using the speech_recognition library.

[0953] Next, this text data is sent to a natural language processing model, which is a generative AI model. The server also contains an emotion engine that analyzes the user's emotions from the received text data. The emotion engine recognizes emotions such as "friendly" or "joyful" and adds this emotional information to the natural language processing model.

[0954] A natural language processing model takes emotion into account to generate an appropriate response text. This response is then converted into audio data on the server and sent to the smart glasses, which then play the response aloud to the user using a bone conduction speaker.

[0955] As a concrete example, consider the case where a user wears smart glasses and says, "Excuse me, where can I find the bread?" while in a store. The smart glasses capture the voice and send it to the server. The server performs speech recognition and converts it into text data, generating the text data "Excuse me, where can I find the bread." This text data is then sent to a natural language processing model by an emotion engine along with a "friendly" emotion. The response generated is "It's near the refrigerated section. Can I help you? You're very welcome." Finally, the smart glasses play this response to the user through a bone conduction speaker.

[0956] Example prompt sentence:

[0957] The user says, "Excuse me, where is the bread?" Please respond with a friendly sentiment.

[0958] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0959] Step 1:

[0960] The user inputs voice through the smart glasses. The microphone built into the smart glasses captures the user's voice. The input is the user's voice data, and the output is the captured voice data. A specific example of this is when the user says, "Excuse me, where is the bread?"

[0961] Step 2:

[0962] The captured audio data is sent to the server in real time. The input is the audio data captured by the smart glasses, and the output is the audio data sent to the server. Specifically, the smart glasses transfer the audio data to the server via Wi-Fi or Bluetooth.

[0963] Step 3:

[0964] The server converts the voice data into text data using the speech_recognition library. The input is the voice data sent to the server, and the output is the text data "Excuse me, where is the bread?". Specifically, the server receives the voice data and executes the process of converting it into text.

[0965] Step 4:

[0966] The server sends this text data to a natural language processing model, which is a generative AI model. The input is the converted text data, and the output is the transmission of the text data to the natural language processing model. Specifically, the server uses an API to send the text data to the model.

[0967] Step 5:

[0968] The emotion engine of the server recognizes the user's emotion from the text data. The input is the text data sent to the natural language processing model, and the output is the emotion recognition data "friendly." In specific operations, the emotion engine of the server analyzes the text data and identifies the emotion.

[0969] Step 6:

[0970] The natural language processing model takes emotion into account and generates a response text. The input is the emotion recognition data "friendly" and the text data "Excuse me, where is the bread?", and the output is the response text "It's near the refrigerated section. Can I help you? You're most welcome." In concrete terms, the natural language processing model takes emotion into account and generates an appropriate response.

[0971] Step 7:

[0972] The server converts the generated response text into voice data. The input is the response text and the output is voice data. Specifically, the server uses voice synthesis technology to convert the text data into voice data.

[0973] Step 8:

[0974] The converted voice data is sent to the smart glasses. The input is the voice data generated by the server, and the output is the voice data sent to the smart glasses. Specifically, the server transfers the voice data to the smart glasses again via Wi-Fi or Bluetooth.

[0975] Step 9:

[0976] The smart glasses use a bone conduction speaker to play audio data to the user. The input is the audio data sent to the smart glasses, and the output is the audio heard by the user. Specifically, the bone conduction speaker of the smart glasses plays the audio to the user, saying, "You're near the refrigerated section. Can you help me? You're very welcome."

[0977] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0978] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0979] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0980] [Fourth embodiment]

[0981] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0982] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0983] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0984] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0985] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0986] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0987] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0988] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0989] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0993] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0994] The system of the present invention converts voice input into text data, generates an appropriate response using a natural language processing model, and then converts that response back into voice data, thereby enabling smooth communication between visually impaired and hearing impaired people, or between people with and without disabilities.

[0995] In this system, the terminal is equipped with a microphone to capture voice input from the user, and the captured voice data is transmitted to the server in real time. The server then uses voice recognition technology to convert the voice input into text data.

[0996] The server then sends the converted text data to a natural language processing model (e.g., a large-scale language model) to generate an appropriate response text, which is then processed again within the server and converted into voice data using speech synthesis technology.

[0997] The terminal then plays the converted audio data back to the user, allowing visually and hearing impaired people to receive information in a format that is convenient for them.

[0998] As a specific example, consider a scenario in which visually impaired user A says, "Hello, how are you?"

[0999] 1. The device captures user A's voice with a microphone and sends the voice data to the server.

[1000] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[1001] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you."

[1002] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[1003] 5. The terminal plays the audio data to User A.

[1004] On the other hand, consider the case where hearing-impaired user B inputs and receives information in text format.

[1005] 1. User B enters "It's a nice day today" in text format.

[1006] 2. The terminal sends the entered text data to the server.

[1007] 3. The server sends the text data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[1008] 4. The server sends the response text to the terminal, which displays it to User B.

[1009] This allows the visually impaired to communicate through speech, and the hearing impaired to communicate through text. By utilizing speech recognition, natural language processing, and speech synthesis technologies at each stage, instant and smooth information exchange becomes possible. This invention enables people with and without disabilities to work and study in the same workplace or school, realizing a more inclusive and equal society.

[1010] The processing flow will be explained below.

[1011] Step 1:

[1012] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[1013] Step 2:

[1014] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[1015] Step 3:

[1016] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[1017] Step 4:

[1018] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[1019] Step 5:

[1020] The server sends the converted text data to a natural language processing model to generate an appropriate response text. For example, a natural language processing model can be used to generate a response such as "Yes, I'm fine, thanks to you."

[1021] Step 6:

[1022] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[1023] Step 7:

[1024] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[1025] Step 8:

[1026] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[1027] Step 9:

[1028] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[1029] Step 10:

[1030] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[1031] Step 11:

[1032] The server sends the received text data to a natural language processing model to generate an appropriate response text, such as "Yes, I feel like going for a walk."

[1033] Step 12:

[1034] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[1035] Step 13:

[1036] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[1037] This series of processes realizes a system that allows smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities.

[1038] Example 1

[1039] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1040] The problem that this invention aims to solve is to realize smooth, real-time communication between users with different disabilities, such as the visually or hearing impaired, and between users without disabilities. In particular, the object is to provide a system that integrates voice input, voice recognition, natural language processing, and voice synthesis technologies, allowing users to receive and send information in a format that is suitable for them.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1042] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text and playing it back to the user, thereby enabling smooth communication between the visually impaired and the hearing impaired, and between users with no disabilities.

[1043] "User" refers to a person who uses the system to input voice or text and receives a response.

[1044] "Audio input" refers to audio signals provided by a user to a system through a microphone device.

[1045] "Text data" refers to text information generated by voice recognition technology or text input.

[1046] "Natural language processing model" refers to a machine learning algorithm for generating appropriate response text from text data.

[1047] "Response text" is the response content generated by the natural language processing model expressed as text information.

[1048] "Voice data" refers to a voice signal converted from response text using voice synthesis technology.

[1049] "Speech recognition technology" refers to technology for converting voice data into text data.

[1050] "Speech synthesis technology" refers to technology for converting text data into natural-sounding speech.

[1051] "Microphone device" refers to a device for capturing a user's voice input.

[1052] A "generative AI model" refers to an algorithm that performs natural language processing using advanced machine learning techniques such as large-scale language models.

[1053] "Communication protocol" refers to the rules and procedures used to send and receive data.

[1054] The present invention is a system that enables smooth communication between visually impaired and hearing impaired people, or between users with and without disabilities, by converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data.

[1055] The device is equipped with a microphone device to capture voice input from the user. When the user speaks, the device's microphone converts the voice into digital audio data. The captured audio data is transmitted to the server in real time. This data transmission uses a secure and fast communication protocol (e.g., HTTPS or WebSocket).

[1056] The server converts the voice data into text data using speech recognition technology. Specifically, it uses a speech recognition engine (e.g., a general speech recognition API). The converted text data is sent to a natural language processing model (e.g., a large-scale language model). The natural language processing model generates an appropriate response text from the text data.

[1057] The generated response text is converted back into audio data using speech synthesis technology. A speech synthesis engine (for example, a general speech synthesis API) is used. This audio data is then sent back to the device and played back to the user. This allows visually and hearing impaired people to receive information in a format that is suitable for them.

[1058] Specific examples

[1059] Scenario in which visually impaired user A speaks

[1060] 1. The device captures user A's voice saying "Hello, how are you?" and sends the voice data to the server.

[1061] 2. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?"

[1062] 3. The server sends the converted text data to a natural language processing model and generates the response text "Yes, I'm fine, thanks to you."

[1063] 4. The server converts the response text into voice data and sends the voice data to the terminal.

[1064] 5. The terminal plays the converted voice data to User A.

[1065] Scenario where hearing impaired user B inputs information in text format

[1066] 1. User B enters "It's a nice day today" in text format.

[1067] 2. The terminal sends the entered text data to the server.

[1068] 3. The server sends the input text data to a natural language processing model and generates the response text "Yes, I feel like going for a walk."

[1069] 4. The server sends the generated response text to the terminal, which displays it to User B.

[1070] Prompt Sentence Examples

[1071] Case of visually impaired user A:

[1072] User A says "Hello, how are you?" Generate an appropriate response to this.

[1073] Deaf User B's Case:

[1074] User B has entered the text "It's a nice day today." Generate an appropriate response to this.

[1075] This system is designed to enable smooth communication between visually and hearing impaired people, as well as users with no disabilities, and makes use of voice recognition, natural language processing, and voice synthesis technologies at each stage to enable instant and smooth information exchange.

[1076] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The device uses a microphone device to capture voice input from the user. When the user says "Hello, how are you?", the device converts the voice into digital voice data. The input is the user's voice, and the output is digital voice data. This voice data is saved in a format such as PCM.

[1079] Step 2:

[1080] The device transmits the captured audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket). The input is the captured digital audio data, and the output is the data sent to the server.

[1081] Step 3:

[1082] The server converts the received voice data into text data using voice recognition technology. Specifically, the conversion is performed using a voice recognition engine. The input is digital voice data, and the output is text data such as "Hello, how are you?" The voice recognition engine analyzes the voice data and converts it into text information.

[1083] Step 4:

[1084] The server sends the converted text data to a natural language processing model. Specifically, it uses a large-scale language model to generate an appropriate response text from the text data. The input is the text data "Hello, how are you?", and the output is the response text "Yes, thanks to you, I'm fine."

[1085] Step 5:

[1086] The server converts the generated response text into voice data. It uses voice synthesis technology to convert text data into voice. Specifically, it uses a voice synthesis engine. The input is the response text "Yes, I'm fine, thanks to you," and the output is digital voice data.

[1087] Step 6:

[1088] The server transmits the converted voice data to the terminal using a secure communication protocol. The input is digital voice data, and the output is data to be transmitted to the terminal.

[1089] Step 7:

[1090] The terminal plays the received audio data, and the user can listen to the audio through the speaker. The input is the digital audio data sent from the server, and the output is the audio played to the user.

[1091] This series of processes enables smooth communication between the visually impaired, the hearing impaired, and users without disabilities. Specifically, the process combines appropriate technologies for each step: voice capture, data transmission, voice recognition, natural language processing, voice synthesis, and voice playback.

[1092] (Application example 1)

[1093] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1094] Visually and hearing impaired people face challenges in communicating smoothly in virtual stores. In particular, they face difficulties in sharing information with store clerks and other customers in a timely manner, which often leads to stress and inconvenience in the process of obtaining specific information or services. Furthermore, conventional systems do not adequately achieve real-time voice input, text conversion, or appropriate response generation.

[1095] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1096] In this invention, the server includes means for acquiring voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for supporting communication between customers and shop staff in a virtual store, means for receiving voice input from the user in real time via a smart device and generating an appropriate response, and means for presenting the response generated by the above means to the user. This enables visually impaired and hearing impaired people to communicate smoothly and in real time using voice and text in a virtual store.

[1097] "User" refers to a person or entity that uses the System.

[1098] "Means for obtaining voice input" refers to the technology or device that captures the voice uttered by the user and that the system captures the voice data.

[1099] "Means for converting into text data" refers to a technology or device for converting acquired voice data into text information.

[1100] "Natural language processing model" refers to a machine learning model used to analyze input text data and generate an appropriate response.

[1101] "Response text" refers to text data of a response generated by a natural language processing model.

[1102] "Means for converting into voice data" refers to a technology or device for converting response text into voice data.

[1103] "Means for playing audio data to a user" refers to a technique or device for letting a user hear the generated audio data.

[1104] A "virtual store" refers to a virtual shopping or service space that is developed online.

[1105] "Customer" refers to a user who uses the virtual store.

[1106] "Shop staff" refers to a person or AI avatar set up to interact with customers within a virtual store.

[1107] "Smart device" refers to a device (e.g., smartphone, smart glasses, etc.) that has internet connectivity and can use a wide variety of applications.

[1108] "Means for generating an appropriate response" refers to a technique or device for generating a response to a user's input using a natural language processing model.

[1109] "Means for presenting to the user" refers to a technology or device that visually or audibly displays to the user the response generated through the smart device.

[1110] This invention is a system that supports smooth communication within a virtual store for visually and hearing impaired people. This system allows users to exchange information within the store via voice and text through their own smart devices.

[1111] 1. Program Overview

[1112] The system includes the following elements:

[1113] Acquire voice input: Acquire voice input from the user.

[1114] Speech recognition: Converts voice input into text data.

[1115] Natural Language Processing: Sending text data to a natural language processing model to generate a response.

[1116] Speech synthesis: Converts response text into audio data.

[1117] Audio playback: The generated audio data is played back to the user.

[1118] Text input and response reception: Send text data to the natural language processing model and receive a response.

[1119] Communication support in virtual stores: Supports communication between customers and store staff within virtual stores.

[1120] 2. System configuration

[1121] Voice input acquisition: Voice input is acquired through the microphone of the smart device (smart glasses or smartphone).

[1122] Speech recognition: Uses the Google Speech Recognition API to convert voice data into text data.

[1123] Natural Language Processing: Uses a Transformer model (e.g., rinna / japanese-gpt-1b) to generate appropriate responses to input text.

[1124] Speech synthesis: Use gTTS (Google Text-to-Speech) to convert the generated response text into voice data.

[1125] Audio playback: Use Pygame to play audio data and present it to the user.

[1126] Text input and response reception: The system sends text input to a natural language processing model and receives the generated response, which it displays to the user.

[1127] Communication support in virtual stores: Receives voice input from users in real time via smart devices and generates and presents appropriate responses.

[1128] 3. Specific examples of processing

[1129] As a concrete example, consider a visually impaired person looking for product information in a virtual store:

[1130] 1. A visually impaired person asks the smart glasses, "Tell me about this product."

[1131] 2. The microphone in the smart glasses captures the audio and converts it to text via the Google Speech Recognition API.

[1132] 3. The text data is sent to a natural language processing model (rinna / japanese-gpt-1b), which generates the response, "This product is a state-of-the-art smartphone. Its features include a high-resolution camera and a large-capacity battery."

[1133] 4. The generated response text is converted to speech using gTTS.

[1134] 5. The audio data is played to the user using Pygame, and the user receives information through audio.

[1135] Example prompt sentence:

[1136] "Prompt: User has asked a question about a product. Please provide a brief description of the relevant product information."

[1137] In this way, the system embodying the present invention enables visually impaired and hearing impaired people to smoothly obtain information and communicate within a virtual store.

[1138] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1139] Step 1:

[1140] The device captures voice input from the user using a microphone. The input is voice data and prepares it to be sent to a speech recognition API. Specifically, the microphone captures the user's voice, converts the data into a digital format, and stores it in memory.

[1141] Step 2:

[1142] The device sends the captured voice data to the Google Speech Recognition API. The input is digital voice data, and the output is corresponding text data. Specifically, the digital voice data is sent to the API endpoint, which performs real-time speech-to-text conversion.

[1143] Step 3:

[1144] The server receives text data from the Google Speech Recognition API and sends it to the natural language processing model. The input is the converted text data, and the output is the generated response text. Specifically, the text data is sent to the API of the natural language processing model, which generates appropriate text as a response.

[1145] Step 4:

[1146] The server sends the generated response text to the Google Text-to-Speech (gTTS) API and converts it into voice data. The input is the response text and the output is voice data. Specifically, the response text is sent to the gTTS API, which uses speech synthesis technology to convert the text into voice.

[1147] Step 5:

[1148] The device plays the generated audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to play the audio data, and the audio is output through the device's speaker.

[1149] Step 6:

[1150] The user inputs text data arbitrarily. The input is text data, which is then prepared for transmission to the natural language processing model. Specifically, the text data is input using the device's keyboard or voice recognition function.

[1151] Step 7:

[1152] The server sends the input text data to the natural language processing model and receives a response. The input is text data and the output is response text. Specifically, the server sends text data to the natural language processing model and receives appropriate text as a response.

[1153] Step 8:

[1154] The server converts the received response text into voice data. The input is the text data to be converted, and the output is voice data. Specifically, the server uses the gTTS API to convert text data into voice data.

[1155] Step 9:

[1156] The device plays the converted audio data to the user. The input is audio data, and the output is audio that the user can hear. Specifically, audio player software (e.g., Pygame) is used to output the audio through the device's speakers.

[1157] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1158] This invention is a communication support system that combines an emotion engine, which converts voice input into text data, generates an appropriate response using a natural language processing model, and converts that response into voice data, thereby realizing smooth communication between the visually impaired and the hearing impaired, or between people with and without disabilities. Furthermore, by using the emotion engine, emotions can be added to the user's utterances, further supporting natural communication.

[1159] In this system, the terminal is equipped with a microphone for capturing voice input from the user and acquires the voice input. The captured voice data is sent to a server in real time. The server uses voice recognition technology to convert the voice input into text data. The converted text data is then sent to an emotion engine, which recognizes the user's emotion. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), which generates a response text. The generated response text is again processed within the server and converted into voice data using voice synthesis technology. The terminal plays the converted voice data to the user. In this way, visually and hearing impaired people can receive information in a format that is suitable for them.

[1160] As a specific example, suppose visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data, "Hello, how are you?" The emotion engine then recognizes user A's emotion from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text, "Yes, I'm fine, thanks to you." At this time, a response is generated that includes the "friendly" emotion. The server then converts the response text into voice data and sends the voice data to the device. The device plays the voice data to user A.

[1161] Meanwhile, consider the case where hearing-impaired User B responds by text input. User B inputs "It's a nice day today, isn't it?" in text format. The device sends the input text data to the server. The server sends the text data to the emotion engine, which recognizes the emotion "joy" from User B's input. The server sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk." The server then sends the response text back to the device in text format. The device displays the received text data to User B.

[1162] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.The present invention aims to improve the quality of communication, enable people with disabilities to share information equally, and reduce social isolation.

[1163] The processing flow will be explained below.

[1164] Step 1:

[1165] The user begins speaking. For example, visually impaired user A might say, "Hello, how are you?"

[1166] Step 2:

[1167] The device uses a microphone to capture the voice of user A. The captured voice data is temporarily saved.

[1168] Step 3:

[1169] The terminal transmits the voice data to the server over the Internet, using an appropriate communication protocol.

[1170] Step 4:

[1171] The server converts the received voice data into text data using voice recognition technology. For example, it uses a voice recognition API to convert the voice into text such as "Hello, how are you?"

[1172] Step 5:

[1173] The server sends the converted text data to an emotion engine to recognize the user's emotion, for example, using the emotion engine to recognize the "friendly" emotion.

[1174] Step 6:

[1175] The server sends the text data and the recognized emotion data to a natural language processing model to generate an appropriate response text. For example, the natural language processing model generates a response such as "Yes, thanks to you, I'm fine."

[1176] Step 7:

[1177] The server converts the generated response text into voice data using speech synthesis technology. For example, it uses a speech synthesis API to create a voice file saying, "Yes, thanks to you, I'm fine."

[1178] Step 8:

[1179] The server sends the generated voice data back to the device, which transmits the voice data using an appropriate communication protocol.

[1180] Step 9:

[1181] The device plays the received voice data, and User A hears the voice response, "Yes, I'm fine, thanks to you."

[1182] Step 10:

[1183] User B (hearing impaired) enters a response in text form, for example, "It's a nice day today."

[1184] Step 11:

[1185] The terminal transmits the text data entered by User B to the server using an appropriate communication protocol.

[1186] Step 12:

[1187] The server sends the received text data to the emotion engine and recognizes the emotion of User B. For example, it recognizes the emotion of "joy."

[1188] Step 13:

[1189] The server sends the text data and recognized emotion data to a natural language processing model, which generates the response text "Yes, I feel like going for a walk."

[1190] Step 14:

[1191] The server then sends the generated response text back to the terminal in text format, with the text data being transmitted using the appropriate communications protocol.

[1192] Step 15:

[1193] The device displays the received text data to User B. User B can read the text, "Yes, I feel like going for a walk."

[1194] This series of processes realizes a system that allows visually impaired and hearing impaired people, or people with and without disabilities, to communicate naturally and with emotion.

[1195] Example 2

[1196] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1197] The present invention aims to enable smooth communication between users who have communication difficulties, such as the visually or hearing impaired. In particular, by combining technologies of speech recognition, emotion recognition, natural language processing, and speech synthesis, the system generates natural responses by adding emotion to user utterances, supporting two-way communication. Another objective is to provide a system that allows information to be transmitted seamlessly even when users use different input methods (voice or text).

[1198] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1199] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to an emotion recognition engine and acquiring emotion data, means for transmitting the emotion data and the text data to a natural language processing model and generating a response text, means for converting the response text into voice data, means for playing the voice data to the user, means for inputting the text data, transmitting it to the natural language processing model and receiving a response, and means for receiving the response text generated by the seventh means, converting it into voice data, and playing it back to the user. This enables visually impaired and hearing impaired people to communicate naturally with emotions.

[1200] "Voice input" is the act or means of obtaining a user's spoken words as a digital signal.

[1201] "Text data" is a data format in which voice input is converted into text information.

[1202] An "emotion recognition engine" is software or a system for analyzing and identifying user emotions from text data.

[1203] A "natural language processing model" is a machine learning model that analyzes given text data and generates responses based on grammar and context.

[1204] "Response text" is text data generated in response to a user input.

[1205] "Audio data" is a data format in which text data is converted into a digital audio signal.

[1206] "Voice recognition technology" is a technology for converting voice input into text data.

[1207] "Speech synthesis technology" is a technology for converting text data into voice data.

[1208] "User" refers to an individual or entity that uses this system, and includes the visually impaired and the hearing impaired.

[1209] This invention is a communication support system that incorporates an emotion recognition engine. By converting voice input into text data, generating an appropriate response using a natural language processing model, and then converting that response back into voice data, it enables smooth communication between the visually impaired, the hearing impaired, and people with and without disabilities. Furthermore, by using an emotion recognition engine, it is possible to assign emotions to the user's speech, supporting more natural communication.

[1210] This system consists of the following main hardware and software components: First, the device is equipped with a microphone that captures voice input from the user. This voice data is sent to a server in real time. The server converts the voice data into text data using voice recognition technology (e.g., a voice recognition API). Next, the converted text data is sent to an emotion recognition engine (e.g., an emotion recognition API), which analyzes the user's emotions. The text data, including the emotion recognition results, is sent to a natural language processing model (e.g., a large-scale language model), and a response text is generated. The generated response text is again processed within the server and converted into voice data using voice synthesis technology (e.g., a voice synthesis API). The device then plays the converted voice data to the user.

[1211] As a specific example, consider the case where visually impaired user A says, "Hello, how are you?" The device captures user A's voice with a microphone and sends the voice data to the server. The server uses voice recognition technology to convert the voice data into text data such as "Hello, how are you?" The emotion recognition engine then recognizes user A's emotions from the text data and detects the "friendly" emotion. The server then sends the text data and emotion data to a natural language processing model, which generates the response text "Yes, I'm fine, thanks to you." The "friendly" emotion is also taken into consideration at this time. The server then converts the response text into voice data and sends the voice data to the device. The device then plays the voice data to user A.

[1212] On the other hand, consider the case where hearing-impaired user B inputs "It's a nice day today." User B inputs in text format, and the device sends the text data to the server. The server sends the text data to an emotion recognition engine, which recognizes the emotion "joy" from user B's input. The server sends the text data and emotion data to a natural language processing model, which generates a response text: "Yes, I feel like going for a walk." The server then sends the generated response text back to the device in text format, and the device displays the received text data to user B.

[1213] Example prompt sentence:

[1214] 1. "When visually impaired person A says, 'Hello, how are you?' Please generate a response text. Please make it a little friendly."

[1215] 2. "Generate a response to the user's input 'What a nice day today!' with the emotion 'Joy'."

[1216] This series of processes enables all users, including those with visual or hearing impairments, to communicate naturally and with emotion.

[1217] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1218] Step 1:

[1219] The user speaks into the microphone, for example, "Hello, how are you?" This is the voice input to the system.

[1220] Input: Speech data ("Hello, how are you?")

[1221] Output: Captured audio data

[1222] Step 2:

[1223] The device uses a microphone to capture the user's voice and obtains the voice data in real time.

[1224] Input: Speech data ("Hello, how are you?")

[1225] Output: Digital audio data

[1226] Step 3:

[1227] The device then transmits the captured audio data over the internet to a server, which is often encrypted to ensure secure transmission.

[1228] Input: Digital audio data

[1229] Output: Audio data sent to the server

[1230] Step 4:

[1231] The server converts the voice data into text data using voice recognition technology (for example, a voice recognition API), which recognizes the voice as text.

[1232] Input: Audio data

[1233] Output: Text data ("Hello, how are you?")

[1234] Step 5:

[1235] The server sends the text data to an emotion recognition engine (e.g., an emotion recognition API) to analyze the user's emotion.

[1236] Input: Text data ("Hello, how are you?")

[1237] Output: Emotion data ("Friendly")

[1238] Step 6:

[1239] The server sends the text data and emotion data to a natural language processing model (e.g., a large-scale language model) to generate a response text.

[1240] Input: Text data ("Hello, how are you?"), Emotion data ("Friendly")

[1241] Output: Response text ("Yes, I'm fine, thanks to you.")

[1242] Step 7:

[1243] The server converts the generated response text into voice data using a voice synthesis technology (for example, a voice synthesis API).

[1244] Input: Response text ("Yes, I'm fine, thanks to you.")

[1245] Output: Audio data

[1246] Step 8:

[1247] The server then sends the audio data back to the device over the internet, where it is again transferred securely.

[1248] Input: Audio data

[1249] Output: Audio data transferred to the device

[1250] Step 9:

[1251] The terminal plays the received audio data over a speaker to provide feedback to the user.

[1252] Input: Audio data

[1253] Output: Played speech ("Yes, I'm fine, thanks to you.")

[1254] Through the above steps, all users, including those with visual or hearing impairments, can communicate naturally and with emotion.

[1255] (Application example 2)

[1256] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1257] Visually and hearing impaired people face challenges when communicating with others in public places or stores, such as difficulty sending and receiving information smoothly. They also face difficulties in communicating naturally and emotionally, which can lead to a sense of social isolation. Therefore, there is a need for a new system that allows people with disabilities to share information equally and realize natural and smooth dialogue.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1259] In this invention, the server includes means for acquiring a voice input from a user, means for converting the voice input into text data, means for transmitting the text data to a natural language processing model and generating a response text, means for recognizing emotions from the text data using an emotion engine, means for generating the response text to which the emotions have been added, and means for converting the response text into voice data, thereby enabling visually impaired and hearing impaired people to communicate naturally and smoothly in public places such as brick-and-mortar stores.

[1260] "User" refers to a person who uses the system.

[1261] "Voice input" refers to voice data collected from a user's speech through an input device such as a microphone.

[1262] "Voice recognition technology" refers to technology that analyzes voice data and converts its content into text data.

[1263] "Text data" refers to data that has been converted from voice input into a string of characters using voice recognition technology.

[1264] A "natural language processing model" refers to an artificial intelligence model that analyzes text data, understands its meaning, and generates an appropriate response.

[1265] "Response Text" refers to the response string generated by a natural language processing model.

[1266] An "emotion engine" refers to technology that recognizes a user's emotions from text data and analyzes those emotions.

[1267] "Audio data" refers to data that has been converted back into audio format from text data using speech synthesis technology.

[1268] "Speech synthesis technology" refers to technology for outputting text data as voice data.

[1269] "Playing back to the user" means letting the user hear the synthesized voice data via a speaker, earphones, or the like.

[1270] "System" refers to a collection of devices and software that performs a series of processes including voice input acquisition, speech recognition, natural language processing, emotion recognition, and speech synthesis.

[1271] As an embodiment of the present invention, we will explain a system that enables visually impaired and hearing impaired people to communicate smoothly in physical stores. This system uses smart glasses and related terminals owned by users, and a server.

[1272] First, the user inputs voice through the smart glasses. The smart glasses have a built-in microphone that captures the user's voice. The captured voice data is sent to the server in real time. The server then converts the voice data into text data using the speech_recognition library.

[1273] Next, this text data is sent to a natural language processing model, which is a generative AI model. The server also contains an emotion engine that analyzes the user's emotions from the received text data. The emotion engine recognizes emotions such as "friendly" or "joyful" and adds this emotional information to the natural language processing model.

[1274] A natural language processing model takes emotion into account to generate an appropriate response text. This response is then converted into audio data on the server and sent to the smart glasses, which then play the response aloud to the user using a bone conduction speaker.

[1275] As a concrete example, consider the case where a user wears smart glasses and says, "Excuse me, where can I find the bread?" while in a store. The smart glasses capture the voice and send it to the server. The server performs speech recognition and converts it into text data, generating the text data "Excuse me, where can I find the bread." This text data is then sent to a natural language processing model by an emotion engine along with a "friendly" emotion. The response generated is "It's near the refrigerated section. Can I help you? You're very welcome." Finally, the smart glasses play this response to the user through a bone conduction speaker.

[1276] Example prompt sentence:

[1277] The user says, "Excuse me, where is the bread?" Please respond with a friendly sentiment.

[1278] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1279] Step 1:

[1280] The user inputs voice through the smart glasses. The microphone built into the smart glasses captures the user's voice. The input is the user's voice data, and the output is the captured voice data. A specific example of this is when the user says, "Excuse me, where is the bread?"

[1281] Step 2:

[1282] The captured audio data is sent to the server in real time. The input is the audio data captured by the smart glasses, and the output is the audio data sent to the server. Specifically, the smart glasses transfer the audio data to the server via Wi-Fi or Bluetooth.

[1283] Step 3:

[1284] The server converts the voice data into text data using the speech_recognition library. The input is the voice data sent to the server, and the output is the text data "Excuse me, where is the bread?". Specifically, the server receives the voice data and executes the process of converting it into text.

[1285] Step 4:

[1286] The server sends this text data to a natural language processing model, which is a generative AI model. The input is the converted text data, and the output is the transmission of the text data to the natural language processing model. Specifically, the server uses an API to send the text data to the model.

[1287] Step 5:

[1288] The emotion engine of the server recognizes the user's emotion from the text data. The input is the text data sent to the natural language processing model, and the output is the emotion recognition data "friendly." In specific operations, the emotion engine of the server analyzes the text data and identifies the emotion.

[1289] Step 6:

[1290] The natural language processing model takes emotion into account and generates a response text. The input is the emotion recognition data "friendly" and the text data "Excuse me, where is the bread?", and the output is the response text "It's near the refrigerated section. Can I help you? You're most welcome." In concrete terms, the natural language processing model takes emotion into account and generates an appropriate response.

[1291] Step 7:

[1292] The server converts the generated response text into voice data. The input is the response text and the output is voice data. Specifically, the server uses voice synthesis technology to convert the text data into voice data.

[1293] Step 8:

[1294] The converted voice data is sent to the smart glasses. The input is the voice data generated by the server, and the output is the voice data sent to the smart glasses. Specifically, the server transfers the voice data to the smart glasses again via Wi-Fi or Bluetooth.

[1295] Step 9:

[1296] The smart glasses use a bone conduction speaker to play audio data to the user. The input is the audio data sent to the smart glasses, and the output is the audio heard by the user. Specifically, the bone conduction speaker of the smart glasses plays the audio to the user, saying, "You're near the refrigerated section. Can you help me? You're very welcome."

[1297] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1298] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1299] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1300] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1301] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1302] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1303] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1304] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1305] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1306] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1307] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1308] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1309] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1310] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1311] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1312] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1313] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1314] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1315] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1316] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1317] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1318] The following is further disclosed regarding the above embodiment.

[1319] (Claim 1)

[1320] a first means for obtaining speech input from a user;

[1321] second means for converting the speech input into text data;

[1322] a third means for transmitting the text data to a natural language processing model and generating a response text;

[1323] a fourth means for converting the response text into voice data;

[1324] a fifth means for playing the audio data to a user;

[1325] a sixth means for inputting the text data, transmitting the text data to the natural language processing model, and receiving a response;

[1326] A system including seventh means for receiving the response text generated by the sixth means, converting it into audio data, and playing it back to the user.

[1327] (Claim 2)

[1328] 2. The system of claim 1, wherein the second means is a means for converting voice input into text data using voice recognition technology.

[1329] (Claim 3)

[1330] 2. The system according to claim 1, wherein the fourth means is a means for converting text data into voice data using a voice synthesis technique.

[1331] "Example 1"

[1332] (Claim 1)

[1333] means for obtaining speech input from a user;

[1334] means for converting the voice input into text data;

[1335] means for transmitting the text data to a natural language processing model and generating a response text;

[1336] means for converting the response text into voice data;

[1337] means for playing the audio data to a user;

[1338] means for inputting the text data, transmitting it to the natural language processing model, and receiving a response;

[1339] means for receiving said response text and playing it to a user.

[1340] (Claim 2)

[1341] 10. The system of claim 1, wherein the means for obtaining audio input comprises means for capturing audio using a microphone device.

[1342] (Claim 3)

[1343] 2. The system of claim 1, wherein the means for converting the voice input into text data is means for converting the voice input into text data using a voice recognition technique.

[1344] (Claim 4)

[1345] 2. The system according to claim 1, wherein the means for converting the response text into voice data is means for converting text data into voice data using a voice synthesis technique.

[1346] (Claim 5)

[1347] The system according to claim 1, further comprising a means for generating an appropriate response by performing a conversion process of the text data and response text using a generative AI model.

[1348] (Claim 6)

[1349] 2. The system according to claim 1, further comprising means for using a secure communication protocol in transmitting and receiving said audio data.

[1350] "Application Example 1"

[1351] (Claim 1)

[1352] a first means for obtaining speech input from a user;

[1353] second means for converting the speech input into text data;

[1354] a third means for transmitting the text data to a natural language processing model and generating a response text;

[1355] a fourth means for converting the response text into voice data;

[1356] a fifth means for playing the audio data to a user;

[1357] a sixth means for inputting the text data, transmitting the text data to the natural language processing model, and receiving a response;

[1358] a seventh means for receiving the response text generated by the sixth means, converting the response text into voice data, and playing the voice data to the user;

[1359] An eighth means of supporting communication between customers and shop staff within the virtual store;

[1360] a ninth means for receiving voice input from the user through the smart device in real time and generating an appropriate response;

[1361] A system including tenth means for presenting the response generated by the ninth means to a user.

[1362] (Claim 2)

[1363] 2. The system of claim 1, wherein the second means is a means for converting voice input into text data using voice recognition technology.

[1364] (Claim 3)

[1365] 2. The system according to claim 1, wherein the fourth means is a means for converting text data into voice data using a voice synthesis technique.

[1366] "Example 2: Combining Emotion Engines"

[1367] (Claim 1)

[1368] a first means for obtaining speech input from a user;

[1369] second means for converting the speech input into text data;

[1370] a third means for transmitting the text data to an emotion recognition engine and acquiring emotion data;

[1371] a fourth means for transmitting the emotion data and the text data to a natural language processing model and generating a response text;

[1372] a fifth means for converting the response text into voice data;

[1373] sixth means for playing the audio data to a user;

[1374] a seventh means for inputting the text data, transmitting the text data to the natural language processing model, and receiving a response;

[1375] a system including eighth means for receiving the response text generated by the seventh means, converting the response text into audio data, and playing the audio data to the user;

[1376] (Claim 2)

[1377] 2. The system of claim 1, wherein the second means is a means for converting voice input into text data using voice recognition technology.

[1378] (Claim 3)

[1379] 2. The system according to claim 1, wherein the fifth means is a means for converting text data into voice data using a voice synthesis technique.

[1380] "Application example 2 when combining emotion engines"

[1381] (Claim 1)

[1382] a first means for obtaining speech input from a user;

[1383] second means for converting the speech input into text data;

[1384] a third means for transmitting the text data to a natural language processing model and generating a response text;

[1385] a fourth means for recognizing emotions from the text data using an emotion engine;

[1386] a fifth means for generating a response text to which the emotion is added;

[1387] a sixth means for converting the response text into voice data;

[1388] seventh means for playing the audio data to a user;

[1389] an eighth means for inputting the text data, transmitting the text data to the natural language processing model, and receiving a response;

[1390] A system including ninth means for receiving the response text generated by the eighth means, converting it into audio data, and playing it back to the user.

[1391] (Claim 2)

[1392] 2. The system of claim 1, wherein the second means is a means for converting voice input into text data using voice recognition technology.

[1393] (Claim 3)

[1394] 2. The system according to claim 1, wherein the sixth means is a means for converting text data into voice data using a voice synthesis technique. [Explanation of symbols]

[1395] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a first means for obtaining speech input from a user; second means for converting the speech input into text data; a third means for transmitting the text data to a natural language processing model and generating a response text; a fourth means for converting the response text into voice data; a fifth means for playing the audio data to a user; a sixth means for inputting the text data, transmitting the text data to the natural language processing model, and receiving a response; A system including seventh means for receiving the response text generated by the sixth means, converting it into audio data, and playing it back to the user.

2. 2. The system of claim 1, wherein the second means is a means for converting voice input into text data using voice recognition technology.

3. 2. The system according to claim 1, wherein the fourth means is a means for converting text data into voice data using voice synthesis technology.

Images