System

Abstract

A system is provided.SOLUTION: A system comprising: means for obtaining audio data; means for converting the audio data into text data; generative model means for converting the text data into sign language video data; and means for displaying the sign language video data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Deaf people need real-time sign language interpretation at events such as press conferences and lectures, but there is a shortage of sign language interpreters, making it difficult to implement in small group work and everyday life situations. This situation poses a major barrier to deaf people's access to information. To solve this problem, a system is needed that provides real-time sign language interpretation video without the need for a sign language interpreter. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including means for acquiring audio data, means for converting the audio data into text data, means for converting the text data into sign language video data, and means for displaying the sign language video data. Furthermore, by including means for preprocessing the audio data, means for encoding the sign language video data into an appropriate multimedia format, and means for transmitting the sign language video data using a secure communication protocol, it is possible to provide real-time sign language interpretation video more effectively.

[0006] "Audio Data" means audio information in digital form collected through a microphone or other audio input device.

[0007] "Text data" is character information generated based on voice data using voice recognition technology.

[0008] A "generative model" is a trained artificial intelligence model that is used to input text data and generate sign language video data.

[0009] "Sign language video data" is data in the form of animation or video that is generated by a generative model and shows sign language movements.

[0010] "Display means" refers to a device such as a monitor or smart glasses that visually displays sign language video data.

[0011] "Preprocessing" refers to processing the collected audio data, such as noise removal and volume normalization.

[0012] "Encoding" is the process of converting sign language video data into an appropriate multimedia format (e.g., MP4, WebM).

[0013] A "secure communication protocol" is a communication method for securely transmitting data, which is usually encrypted (e.g., HTTPS). [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system that converts audio data into sign language video data in real time and displays it to solve the shortage of sign language interpreters and the difficulty of allocating them. Specifically, sign language interpretation is realized by acquiring audio data, converting it into text data, converting it into sign language video data using a generative model, and finally displaying the sign language video on a display means.

[0036] Program processing

[0037] Audio collection and transmission

[0038] 1. The user starts speaking

[0039] User: Speaks into the microphone. Used in meetings, lectures, and everyday conversations.

[0040] 2. The device collects the audio data

[0041] Terminal: Collects user voice data in digital format in real time via a microphone.

[0042] 3. The device preprocesses the audio data

[0043] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[0044] 4. The device sends the audio data to the server

[0045] Terminal: Sends the pre-processed audio data to the server using a secure communication protocol.

[0046] Analysis of voice data and sign language conversion

[0047] 5. The server receives the audio data

[0048] Server: Receives the audio data sent from the terminal and establishes a data stream.

[0049] 6. The server converts the audio data into text data

[0050] Server: Using voice recognition technology, the received voice data is converted into text data.

[0051] 7. The server processes the text data using natural language processing

[0052] Server: Analyzes the generated text data using a natural language processing algorithm and converts it into a format suitable for sign language.

[0053] 8. The server generates sign language video using the generative model

[0054] Server: Using a trained generative model, it generates animated sign language videos based on text data.

[0055] Sign language video generation and display

[0056] 9. The server encodes the sign language video data

[0057] Server: Encodes the generated sign language video data into an appropriate multimedia format.

[0058] 10. The server sends the sign language video data to the device.

[0059] Server: Sends encoded sign language video data to the terminal using a secure communication protocol.

[0060] 11. The device receives the sign language video data

[0061] Terminal: Receives sign language video data sent from the server and stores it in the display buffer.

[0062] 12. The device displays sign language images

[0063] Terminal: The received sign language video is displayed on a monitor or smart glasses display, allowing the user to visually understand the sign language interpretation in real time.

[0064] Specific examples

[0065] 1. Use in meetings

[0066] User: The meeting facilitator makes a presentation using a microphone.

[0067] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0068] Server: Receives voice data and converts it into text data using speech recognition technology. Generates sign language video data using a generative model, encodes it, and sends it to the device.

[0069] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[0070] 2. Daily communication

[0071] User: Starts speaking to place an order at a cafe.

[0072] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0073] Server: Converts voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[0074] Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[0075] As described above, the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving access to information for the hearing impaired.

[0076] The processing flow will be explained below.

[0077] Step 1:

[0078] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0079] Step 2:

[0080] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0081] Step 3:

[0082] Terminal: Preprocessing such as noise reduction and volume normalization is performed on the collected audio data. Here, signal processing algorithms are used to adjust the audio data so that it is clear.

[0083] Step 4:

[0084] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[0085] Step 5:

[0086] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[0087] Step 6:

[0088] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0089] Step 7:

[0090] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[0091] Step 8:

[0092] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[0093] Step 9:

[0094] Server: Encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces the file size and improves transmission and display efficiency.

[0095] Step 10:

[0096] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[0097] Step 11:

[0098] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[0099] Step 12:

[0100] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand the smooth sign language interpretation.

[0101] As described above, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving information access for the hearing impaired.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] Currently, there is a shortage of sign language interpreters and difficulty in locating them, which poses a major barrier for the hearing impaired, especially in situations where real-time communication is required. In addition, the technology for converting speech information into sign language is still in its infancy, and improvements in accuracy and speed are required. To solve these issues, a system that provides high-quality, real-time sign language interpretation is needed.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes a speech recognition unit that converts speech data into text data, a unit that performs natural language processing on the text data and converts it into a format suitable for sign language video data, and a unit that converts the text data into sign language video data using a generative model, thereby enabling high-quality real-time sign language interpretation.

[0107] The "means for the user to produce voice" is a system component that includes the user inputting voice into a microphone during a meeting or everyday conversation.

[0108] The "means for the terminal to collect the voice as digital voice data" refers to the function of converting analog voice into a digital signal and collecting it in real time through a microphone connected to the terminal.

[0109] The "means for preprocessing the audio data and performing noise removal and volume normalization" refers to software or hardware functions for removing unnecessary noise from the audio data and performing processing to maintain a constant volume.

[0110] The "means for transmitting preprocessed audio data to a server" is a device or method for transferring the preprocessed audio data to a server via a secure communications protocol.

[0111] The "server means for receiving the voice data" is a function of the server for receiving and processing the voice data transmitted from the terminal.

[0112] The "voice recognition means for converting the voice data into text data" refers to a technology or software function for analyzing a voice signal and converting the content into text information.

[0113] "Means for natural language processing the text data and converting it into a format suitable for sign language video data" refers to algorithms or software functions for performing natural language processing on text data, extracting and organizing the information necessary for sign language expression.

[0114] "Means for converting the text data into sign language video data using a generative model" refers to technology or software functions that convert text data into sign language animation video using a generative model.

[0115] "Means for encoding the sign language video data into a suitable multimedia format" refers to software or hardware functionality for compressing and encoding the generated sign language video data into a suitable multimedia format.

[0116] The "means for transmitting the encoded sign language video data to the receiving terminal" refers to a communication means or protocol for securely transferring the encoded sign language video data to the terminal.

[0117] The "means for displaying the received sign language video data" refers to a display device or software function for visually displaying the received sign language video data to the user on the terminal.

[0118] The "means for displaying the sign language video data on a display device" refers to hardware and software functions for displaying the sign language video data on a terminal display or an external monitor.

[0119] "Means for using a microphone and audio processing software to collect and pre-process said audio data" refers to the function of using a microphone to collect audio and pre-processing the collected audio with software.

[0120] This invention is a system that converts audio data into sign language video data in real time and displays it in order to solve the shortage of sign language interpreters and the difficulty of allocating them. How this system is implemented will be explained below in detail.

[0121] First, situations where a user may speak include a meeting or everyday conversation. The user inputs the voice through a microphone. For example, a general microphone is used in a meeting.

[0122] Next, the device, such as a laptop, collects this voice in real time as digital audio data. Specifically, a microphone connected to the device converts analog audio into a digital signal, and the collected audio data is stored in the device.

[0123] The collected voice data is pre-processed within the device, which includes noise reduction and volume normalization. The voice processing software used is, for example, general voice processing software.

[0124] The pre-processed audio data is then sent to the server using a secure communication protocol, TLS / SSL, which encrypts the data during transmission.

[0125] The server receives the voice data sent from the device. For example, a general-purpose server used as part of an online service is used as the server. The server uses the received voice data for analysis as is.

[0126] The server then converts the voice data into text data. This voice recognition process uses, for example, voice recognition technology. Specifically, a voice recognition engine analyzes the voice signal and converts it into text data.

[0127] The generated text data is then subjected to natural language processing and converted into a format suitable for sign language video data. A general natural language processing library is often used as the natural language processing algorithm. This analyzes the structure of the sentence and extracts the information necessary for sign language expression.

[0128] The converted text data is then converted into sign language video data using a generative model, such as a trained generative AI model. At this stage, the text data is converted into sign language animation video.

[0129] The resulting sign language video data is then encoded into a suitable multimedia format, for example using multimedia software, which compresses the video data and optimizes it for easy transmission and playback.

[0130] The encoded sign language video data is securely transmitted from the server to a receiving terminal, such as a smartphone carried by a user.

[0131] The receiving terminal receives the sign language video data and displays it on a display device, allowing the user to visually understand the sign language interpretation in real time.

[0132] Specific examples

[0133] Use in meetings

[0134] 1. User: The meeting facilitator makes a presentation using a standard microphone.

[0135] 2. Terminal: A laptop computer installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0136] 3. Server: Receives the voice data and converts it into text data using speech recognition technology. It then generates sign language video data using a generative model, encodes it, and sends it to the device.

[0137] 4. Terminal: The received sign language video is displayed on the screen in the conference room in real time. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[0138] Daily communication

[0139] 1. User: Starts speaking to place an order at a cafe.

[0140] 2. Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0141] 3. Server: Converts the voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[0142] 4. Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[0143] Prompt Sentence Examples

[0144] "I would like to convert what is being said in a meeting into sign language in real time. Please tell me how to use this system."

[0145] "Please explain the setup procedure for displaying orders in sign language at a cafe."

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] The user speaks.

[0149] During meetings or everyday conversations, users input their voice into a microphone. This voice becomes the input to the system. The voice uttered by the user is sent to the terminal via the microphone.

[0150] Step 2:

[0151] The terminal collects the voice data as digital voice data.

[0152] A microphone connected to the device converts analog voice into a digital signal and collects it in real time. The collected digital voice data is stored in the device's memory. Analog voice data is the input, and digital voice data is obtained as the output.

[0153] Step 3:

[0154] The device preprocesses the audio data.

[0155] The device performs preprocessing on the collected audio data, such as noise reduction and volume normalization. Audio processing software is used to remove noise and maintain a constant volume. The input for preprocessing is digital audio data, and the output is the preprocessed audio data.

[0156] Step 4:

[0157] The terminal transmits the preprocessed audio data to the server.

[0158] The preprocessed audio data is sent to the server using a secure communication protocol (TLS / SSL). At this time, the data is encrypted and sent. The input is the preprocessed audio data, and the output is the data sent to the server.

[0159] Step 5:

[0160] The server receives the audio data.

[0161] The server receives the voice data sent from the terminal. The received voice data is stored in the server's memory. The input is the voice data sent from the terminal, and the output is the received voice data.

[0162] Step 6:

[0163] The server converts the voice data into text data.

[0164] The server uses speech recognition technology to convert the received voice data into text data. The speech recognition engine analyzes the voice signal and generates a corresponding text representation. The input is voice data and the output is text data.

[0165] Step 7:

[0166] The server performs natural language processing on the text data.

[0167] The generated text data is analyzed using natural language processing algorithms and converted into a format suitable for sign language. Sentence structure is analyzed and important information is extracted. The input is text data, and the output is data in a format suitable for sign language video data.

[0168] Step 8:

[0169] The server generates sign language video using the generative model.

[0170] Based on the generated data, a trained generative model is used to generate animated sign language videos. The input is data in a format suitable for sign language video data, and the output is sign language video data.

[0171] Step 9:

[0172] The server encodes the sign language video data.

[0173] The server encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4 format). Multimedia software is used for the encoding process. The input is the sign language video data, and the output is the encoded sign language video data.

[0174] Step 10:

[0175] The server transmits the encoded sign language video data to the terminal.

[0176] The encoded sign language video data is transmitted to a terminal using a secure communication protocol, where the input is the encoded sign language video data and the output is the data transmitted to the terminal.

[0177] Step 11:

[0178] The terminal receives the sign language video data.

[0179] The terminal receives the encoded sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent to the terminal, and the output is the received sign language video data.

[0180] Step 12:

[0181] The device displays the sign language video.

[0182] The terminal displays the received sign language video data on a display device, such as a screen in a conference room or smart glasses. The input is the received sign language video data, and the output is the displayed sign language video.

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Modern autonomous driving technology faces the problem of hearing-impaired people having difficulty understanding instructions from voice assistants. Furthermore, in situations where real-time sign language interpretation is not available, hearing-impaired people have difficulty obtaining information quickly and accurately. This can lead to reduced safety and convenience while driving. Therefore, there is a need for the development of a system that converts instructions from voice assistants into sign language images in real time and displays them through smart glasses.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a generative model means for converting the text data into sign language video data, thereby enabling instructions from the voice assistant to be converted into sign language video in real time and displayed through the smart glasses.

[0188] "Audio data" is a digital representation of sound collected by an acoustic sensor such as a microphone.

[0189] "Text data" is data in which voice data is expressed as character information.

[0190] A "generative model" is an artificial intelligence model that generates new data based on input data.

[0191] "Sign language video data" is video data that visually expresses actions corresponding to sign language.

[0192] A "display means" is a hardware or software device for visually presenting an image to a user.

[0193] "Smart glasses" are eyeglass-like devices worn by the user that can display information on a screen.

[0194] "Preprocessing" refers to performing quality improvement processes such as noise removal and volume normalization on the collected data.

[0195] A "multimedia format" is a digital format for uniformly handling different types of data, such as audio, video, and text data.

[0196] "Encoding" is the process of converting data into a particular format.

[0197] A "server" is a computer system for processing data.

[0198] The present invention is a system for converting voice data into sign language video data in real time and displaying it. This system is intended for use in autonomous vehicles, and displays voice instructions and navigation information from a driving assistance system as sign language video.

[0199] The system of the present invention includes the following hardware and software.

[0200] Hardware and software used

[0201] Hardware: smart glasses, microphones, in-car computer systems

[0202] Software: speech recognition engines, natural language processing libraries, generative AI models (e.g., Transformers), sign language animation generation engines, communication protocols (e.g., HTTPS)

[0203] Program processing

[0204] Imagine a scenario in which a user wearing smart glasses receives instructions from a voice assistant inside a car.

[0205] First, the user hears voice commands from the voice assistant. This voice data is collected by a microphone installed in the vehicle. The collected voice data undergoes pre-processing such as noise reduction and volume normalization to improve the data quality. The pre-processed voice data is then sent to a server via the vehicle's computer system using a secure communication protocol.

[0206] The voice data received by the server is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then analyzed by a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converted into a format suitable for sign language. After that, a generative AI model (e.g., GPT-3) is used to generate sign language video data based on the analyzed text data. This generated sign language video data is then encoded into an appropriate multimedia format and sent back to the in-vehicle system.

[0207] The vehicle's onboard computer system receives the sign language video data sent from the server and displays it on the smart glasses' display, allowing the user to understand voice instructions in sign language in real time.

[0208] Specific examples

[0209] Example 1: An in-car voice assistant instructs the driver to "turn right at the next traffic light." The microphone collects the audio and sends it to the server after preprocessing. The server converts the audio instruction into text and generates sign language using natural language processing. The sign language video data is displayed on the smart glasses, allowing the user to understand the instruction through sign language.

[0210] Example 2: An in-car voice assistant instructs, "Set the vehicle speed to 50 km / h." Similarly, voice data is collected, pre-processed, transmitted, and converted into text data. The generated sign language video data is displayed on the smart glasses, allowing the user to understand the instruction.

[0211] Example prompts to input to the generative AI model

[0212] "Please convert the text "Turn right at the next traffic light" into video data of sign language."

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] The device collects voice instructions using the in-car microphone. Specifically, when an instruction is given from the voice assistant, the microphone converts the analog voice data into a digital format and captures it as a data stream. The input of this step is an analog voice signal, and the output is digital voice data.

[0216] Step 2:

[0217] The digital audio data collected by the device is preprocessed, such as by noise removal and volume normalization. Specifically, an audio processing library (e.g., FFmpeg) is used to improve the quality of the collected data. The input of this step is the digital audio data, and the output is the preprocessed audio data.

[0218] Step 3:

[0219] The terminal transmits the preprocessed voice data to the server via the in-vehicle system using a secure communication protocol (e.g., HTTPS). Specifically, the encoded data is encrypted and transmitted using the HTTPS protocol. The input of this step is the preprocessed voice data, and the output is the data transmitted to the server.

[0220] Step 4:

[0221] The server converts the voice data received from the device into text data using a voice recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice recognition algorithm analyzes the voice characteristics and converts them into a corresponding string of characters. The input of this step is the voice data received by the server, and the output is text data.

[0222] Step 5:

[0223] The server analyzes the generated text data using a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converts it into a format suitable for sign language. Specifically, the text data is semantically analyzed and broken down into phrases that correspond to sign language. The input to this step is text data, and the output is text data in a format suitable for sign language.

[0224] Step 6:

[0225] The server uses a generative AI model (e.g., GPT-3) to generate sign language video data based on the analyzed text data. Specifically, the generative AI model generates appropriate sign language movements and outputs the data as 3D animation. The input for this step is text data in a format suitable for sign language, and the output is sign language video data.

[0226] Step 7:

[0227] The server encodes the generated sign language video data into an appropriate multimedia format. Specifically, it encodes it into a video format (e.g., MP4) and prepares it for transmission. The input of this step is the sign language video data, and the output is the encoded sign language video data.

[0228] Step 8:

[0229] The server then sends the encoded sign language video data to the terminal again using a secure communication protocol. Specifically, the encoded data is encrypted and sent using the HTTPS protocol. The input to this step is the encoded sign language video data, and the output is the data sent to the terminal.

[0230] Step 9:

[0231] The terminal receives the sign language video data sent from the server and displays it on the smart glasses display. Specifically, it decodes the received data and visually displays it to the user. The input of this step is the sign language video data sent from the server, and the output is the sign language video displayed on the smart glasses display.

[0232] Step 10:

[0233] The user visually understands the sign language video displayed through the smart glasses. Specifically, the user looks at the smart glasses display and understands the instructions from the voice assistant through the sign language video. The input for this step is the sign language video displayed on the smart glasses display, and the output is the user's understanding.

[0234] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0235] This system aims to improve information access for the hearing impaired by providing high-quality real-time sign language interpretation even when a sign language interpreter is not available. In particular, by combining it with an emotion engine that recognizes the user's emotions, it realizes sign language interpretation that responds to the user's emotions.

[0236] Program processing

[0237] Audio collection and transmission

[0238] 1. The user starts speaking

[0239] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0240] 2. The device collects the audio data

[0241] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0242] 3. The device preprocesses the audio data

[0243] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[0244] 4. The device sends the audio data to the server

[0245] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[0246] Analysis of voice data and sign language conversion

[0247] 5. The server receives the audio data

[0248] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[0249] 6. The server converts the audio data into text data

[0250] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0251] 7. The server processes the text data using natural language processing

[0252] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[0253] 8. The server generates sign language video using the generative model

[0254] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[0255] 9. The server recognizes the user's emotions using an emotion engine

[0256] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[0257] 10. The server adjusts the sign language video data based on emotions.

[0258] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, if the user expresses anger, the server may adjust the expression by emphasizing the sign language movements.

[0259] 11. The server encodes the sign language video data

[0260] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[0261] 12. The server sends the sign language video data to the device.

[0262] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[0263] 13. The device receives sign language video data

[0264] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[0265] 14. The device displays sign language images

[0266] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[0267] Specific examples

[0268] 1. Use in meetings

[0269] User: The meeting facilitator makes a presentation using a microphone.

[0270] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0271] Server: Receives voice data and converts it into text using speech recognition technology. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the sign language video. After encoding, it is sent to the device.

[0272] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video to understand the content of the presentation and also grasp the presenter's emotions.

[0273] 2. Daily communication

[0274] User: Starts speaking to place an order at a cafe.

[0275] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0276] Server: Converts voice data into text using speech recognition technology and analyzes it using natural language processing algorithms. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the video. After encoding, it is sent to the device.

[0277] Device: The received sign language video is displayed on the smart glasses. Through the video, the hearing-impaired person can understand the order details and also grasp the emotions of the store clerk.

[0278] As described above, the system of the present invention provides high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired and understanding the user's emotions.

[0279] The processing flow will be explained below.

[0280] Step 1:

[0281] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0282] Step 2:

[0283] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0284] Step 3:

[0285] Terminal: The collected audio data is preprocessed using noise reduction and volume normalization to improve the quality of the data. Specifically, the audio signal is processed using a digital filter to reduce background noise.

[0286] Step 4:

[0287] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with Transport Layer Security (TLS).

[0288] Step 5:

[0289] Server: Receives the audio data sent from the device, establishes a data stream, and temporarily stores the received data in local storage.

[0290] Step 6:

[0291] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0292] Step 7:

[0293] Server: The generated text data is analyzed using a natural language processing (NLP) algorithm and converted into a format suitable for sign language. The NLP algorithm performs grammatical and semantic analysis to generate a text format that is optimal for sign language conversion.

[0294] Step 8:

[0295] Server: Converts the analyzed text data into animated sign language videos using a trained generative model. The generative model (e.g., a Transformer model) includes a neural network that inputs text data and generates sign language videos.

[0296] Step 9:

[0297] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[0298] Step 10:

[0299] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, it emphasizes the sign language movements to indicate anger, and softens the sign language movements to indicate joy.

[0300] Step 11:

[0301] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[0302] Step 12:

[0303] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with TLS.

[0304] Step 13:

[0305] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[0306] Step 14:

[0307] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[0308] As a result, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired. Furthermore, by understanding the user's emotions and providing appropriate sign language expressions, the system achieves more natural and empathetic communication.

[0309] Example 2

[0310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] It is difficult for hearing-impaired people to obtain accurate and emotionally rich information in real time even in situations where a sign language interpreter is not available. Furthermore, there are an increasing number of situations where communication that includes emotional nuances, rather than just conveying information, is required. The purpose of this invention is to solve these problems and improve information access for hearing-impaired people.

[0312] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0313] In this invention, the server includes means for acquiring audio data, means for preprocessing the audio data, means for converting the audio data into text data, means for generative modeling for converting the text data into sign language video data, means for adjusting the sign language video data based on a user's emotions, means for encoding the sign language video data into an appropriate multimedia format, and means for displaying the sign language video data, thereby enabling real-time, high-quality sign language interpretation and the generation of sign language video that reflects the user's emotions.

[0314] The "means for acquiring voice data" refers to a device or system that collects voices uttered by a user as digital data.

[0315] "Means for preprocessing audio data" refers to a device or system that performs processes such as noise removal and volume normalization on collected audio data to improve the quality of the data.

[0316] "Means for converting voice data into text data" refers to a device or system that uses voice recognition technology to convert voice data into text information.

[0317] "Generative model means for converting text data into sign language video data" refers to a device or system that uses a generative model algorithm to convert text information into sign language animation video.

[0318] The "means for adjusting sign language video data based on the user's emotions" refers to a device or system that analyzes the user's emotional state and adjusts the movement of the sign language video accordingly.

[0319] A "means for encoding sign language video data into a suitable multimedia format" is a device or system that converts sign language video data into a suitable multimedia format (e.g., MP4) for more efficient transmission and display.

[0320] "Means for displaying sign language video data" refers to a device or system that displays the generated and adjusted sign language video data on a device such as a monitor or smart glasses.

[0321] A "means for transmitting audio data using a secure communication protocol" is a device or system that transmits captured and pre-processed audio data to a server in a secure and reliable manner (e.g., HTTPS).

[0322] "Means for recognizing user emotions from sign language video data using an emotion engine" refers to a set of algorithms and devices for analyzing and recognizing user emotions from audio data and text data.

[0323] MODE FOR CARRYING OUT THE INVENTION

[0324] This invention provides a system that provides high-quality real-time sign language interpretation even when a sign language interpreter is not available, improving information access for the hearing impaired. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to realize sign language interpretation that responds to emotions.

[0325] Hardware and Software

[0326] Hardware:

[0327] 1. Device:

[0328] High-sensitivity microphones: used to collect audio precisely. An example is the Shure SM58.

[0329] Display: Use a monitor or smart glasses (e.g., Google Glass) to visually display the sign language image.

[0330] 2. Server:

[0331] Use servers with high-performance computing resources, such as cloud-based servers (e.g., AWS, Google Cloud).

[0332] software:

[0333] 1. Device:

[0334] Audio pre-processing software: Use the noise reduction feature in Audacity.

[0335] Communication Protocol: HTTPS and TLS are used to ensure secure communication.

[0336] 2. Server:

[0337] Speech recognition technology: Google Cloud Speech-to-Text API.

[0338] Natural language processing algorithms: SpaCy, NLTK.

[0339] Generative model: Generates sign language animation video using a Transformers model.

[0340] Sentiment Engine: IBM Watson sentiment analysis API.

[0341] Encoding tool: Use FFmpeg to encode the sign language video data into MP4 format.

[0342] Overview of program processing

[0343] 1. User:

[0344] Speak into the microphone. The system can be used in a variety of situations, such as meetings and everyday conversations.

[0345] 2. Terminal:

[0346] Collect user voice data in real time through a microphone.

[0347] Preprocess the audio data using Audacity's noise reduction function.

[0348] The preprocessed audio data is encrypted with TLS using HTTPS and sent to the server.

[0349] 3. Server:

[0350] A data stream is established to receive and temporarily store audio data sent from the terminal.

[0351] Convert the audio data into text data using the Google Cloud Speech-to-Text API.

[0352] Using SpaCy and NLTK, text data is processed using natural language processing and converted into a format suitable for sign language.

[0353] Generate animated sign language videos using Transformers models.

[0354] Using IBM Watson's sentiment analysis API, it recognizes user emotions from voice and text data.

[0355] Adjust sign language video data based on recognized emotions, for example, enhancing sign movements to indicate anger.

[0356] Use an encoding tool (FFmpeg) to encode the adjusted sign language video data into MP4 format.

[0357] The encoded sign language video data is sent to the terminal using HTTPS.

[0358] 4. Terminal:

[0359] The sign language video data transmitted from the server is received and stored in the display buffer.

[0360] The received sign language video is displayed on a monitor or smart glasses display.

[0361] Specific examples

[0362] Meeting use:

[0363] A user acts as the meeting facilitator and makes a presentation using a microphone. A device (e.g., a laptop PC) installed in the conference room collects audio in real time, preprocesses it, and sends it to a server. The server converts the received audio data into text using the Google Cloud Speech-to-Text API, and generates a sign language video using a generative model and emotion engine. The video data is encoded, sent back to the device, and displayed on a screen in the conference room. Hearing-impaired participants can understand the content of the speech and the speaker's emotions through this sign language video.

[0364] Daily communication:

[0365] A user begins speaking to place an order at a cafe. The smartphone collects the audio, preprocesses it using Audacity, and sends the audio data to a server. The server converts the audio data into text using the Google Cloud Speech-to-Text API and creates a sign language video using a generative model and emotion engine. The adjusted and encoded data is sent back to the device and displayed on the smart glasses display. The hearing-impaired user can understand the order and the waiter's emotions through this video.

[0366] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0367] Step 1:

[0368] The user speaks

[0369] A user speaks into a microphone for meetings or everyday conversations. The input is the user's spoken voice. The output is an analog audio signal.

[0370] Step 2:

[0371] The device collects the audio

[0372] The device uses a high-sensitivity microphone (e.g., Shure SM58) to collect the user's voice as a digital signal. The input is an analog audio signal. The output is digital audio data.

[0373] Step 3:

[0374] The device preprocesses the audio

[0375] The device uses Audacity's noise reduction function to remove noise from the collected audio data and normalizes the volume. The input is digital audio data. The output is pre-processed, clear audio data.

[0376] Step 4:

[0377] The device sends the audio data to the server.

[0378] The device sends the preprocessed audio data to the server using HTTPS, and the TLS protocol is used for data encryption. The input is the preprocessed audio data. The output is the audio data sent to the server via a secure communication protocol.

[0379] Step 5:

[0380] The server receives the audio data.

[0381] The server receives the voice data sent from the terminal and stores it in a temporary data stream, which is used for subsequent processing. The input is the voice data sent from the terminal. The output is the temporarily stored voice data.

[0382] Step 6:

[0383] The server converts the audio data into text

[0384] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data. The output is a highly accurate text data.

[0385] Step 7:

[0386] The server processes the text in natural language.

[0387] The server analyzes the generated text data using natural language processing algorithms such as SpaCy and NLTK and converts it into a format suitable for sign language. The input is text data. The output is data in a format suitable for sign language.

[0388] Step 8:

[0389] The server generates the sign language video

[0390] The server uses a Transformers model to convert natural language processed text data into sign language animation. The input is data in a format suitable for sign language. The output is sign language animation data.

[0391] Step 9:

[0392] The server recognizes emotions

[0393] The server uses IBM Watson's sentiment analysis API to recognize the user's emotions from voice and text data. The input is voice and text data. The output is the recognized emotion information.

[0394] Step 10:

[0395] The server adjusts the sign language video based on emotions.

[0396] The server adjusts the sign language video data based on the recognized emotion. For example, if the emotion is "sadness," the sign language movements are slowed down. The input is the sign language video data and the emotion information. The output is the adjusted sign language video data.

[0397] Step 11:

[0398] The server encodes the sign language video

[0399] The server encodes the adjusted sign language video data into MP4 format using FFmpeg. The input is the adjusted sign language video data. The output is the encoded sign language video data in MP4 format.

[0400] Step 12:

[0401] The server sends the sign language video data

[0402] The server sends the encoded sign language video data to the terminal using HTTPS to ensure reliability and security of the communication. The input is the encoded sign language video data. The output is the sign language video data sent to the terminal.

[0403] Step 13:

[0404] The device receives the sign language video data.

[0405] The terminal receives sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent from the server. The output is the sign language video data stored in the display buffer.

[0406] Step 14:

[0407] The device displays a sign language video.

[0408] The device displays the received sign language video on a monitor or smart glasses (e.g., Google Glass). The input is the sign language video data stored in the display buffer. The output is the sign language video displayed on the monitor or smart glasses.

[0409] (Application example 2)

[0410] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0411] Current sign language interpretation systems have difficulty reflecting the user's emotions appropriately, making it difficult to convey emotional nuances. Furthermore, it is difficult to provide high-quality real-time sign language interpretation when a sign language interpreter is unavailable. This makes it difficult for the hearing impaired to accurately understand information.

[0412] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0413] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for generating a generative model for converting the text data into sign language video data, means for analyzing a user's emotions, means for adjusting the sign language video data based on the user's emotions, and means for displaying the sign language video data. This enables high-quality real-time sign language interpretation that reflects the user's emotions even in places where a sign language interpreter is not available, enabling people with hearing impairments to accurately understand information.

[0414] The "means for acquiring voice data" refers to a means for collecting the voice uttered by the user in digital form.

[0415] The "means for converting voice data into text data" refers to a means for converting collected voice data into text information using natural language processing technology.

[0416] The "generative model means for converting text data into sign language video data" is a means for simulating sign language movements based on character information and generating the results as video data.

[0417] The "means for displaying sign language video data" refers to a means for displaying the generated sign language video on a display device such as a monitor or smart glasses.

[0418] "Means for analyzing user emotions" refers to means for reading and analyzing the user's emotional state from voice data or text data.

[0419] The "means for adjusting sign language video data based on the user's emotions" refers to a means for adjusting the movements and expressions of the sign language video to reflect the analyzed emotions of the user.

[0420] The present invention is a system that acquires voice data, analyzes the voice data, and converts it into sign language video data in real time. Furthermore, it can realize natural and emotional communication by identifying the user's emotions and adjusting the sign language video according to those emotions. The following hardware and software are used to implement the present invention.

[0421] Hardware used

[0422] 1. Audio collection device: Use smart glasses or a smartphone equipped with a highly sensitive microphone.

[0423] 2. Display device: Use smart glasses or a monitor equipped with a display for displaying sign language images.

[0424] 3. Data processing server: A high-performance server is used to process and manage audio data and sign language video data.

[0425] Software used

[0426] 1. Speech recognition software: Uses a speech recognition API to convert voice data into text data.

[0427] 2. Natural Language Processing Algorithms: Use natural language processing (NLP) algorithms to analyze text data.

[0428] 3. Emotion Recognition Engine: An emotion recognition engine is used to analyze user emotions from voice and text.

[0429] 4. Sign language generative model: Use a generative AI model (e.g., Transformer model) that converts text into sign language.

[0430] 5. Video Encoder: Use multimedia software to encode the generated sign language video data into an appropriate format.

[0431] System Functions and Flow

[0432] 1. Audio collection: The microphone installed on the device is used to acquire the user's audio data.

[0433] 2. Speech recognition: The collected voice data is sent to the server and converted into text data using a speech recognition API.

[0434] 3. Sentiment analysis: Recognize user emotions from text data and original audio data.

[0435] 4. Sign Language Generation: Use a generative AI model to convert text data into sign language video data.

[0436] 5. Emotional adjustment: Adjust the expression of sign language video based on the recognized emotional information.

[0437] 6. Video display: After buffering, the adjusted sign language video data is displayed on the display device.

[0438] Specific examples

[0439] In a physical store, when a cashier asks a customer, "Do you have a point card?", this system allows the cashier to display a sign language image through the smart glasses. Furthermore, if the emotion of "happiness" is analyzed from the customer's attitude and facial expression, the sign language image is adjusted to reflect that emotion, resulting in more natural and human-like communication.

[0440] Prompt Sentence Examples

[0441] "Real-time sign language interpretation when a cashier asks, 'Do you have a loyalty card?'"

[0442] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0443] Step 1:

[0444] (Audio collection)

[0445] The device captures the user's voice in real time through a microphone. The input is the user's speech, and the output is digital voice data. The device uses a highly sensitive microphone to improve the accuracy of the collected voice data.

[0446] Step 2:

[0447] (Preprocessing of audio data)

[0448] The device performs preprocessing on the collected voice data, such as noise removal and volume normalization, to improve the quality of the data. The input is the raw voice data acquired in step 1, and the output is clear, preprocessed voice data. This processing improves the accuracy of voice recognition.

[0449] Step 3:

[0450] (Sending audio data)

[0451] The device sends the preprocessed audio data to the server. The input is the preprocessed audio data, and the output is a secure data transfer to the server. HTTPS is used as the communication protocol, and the data is transmitted encrypted.

[0452] Step 4:

[0453] (Speech recognition of voice data)

[0454] The server converts the received voice data into text data using a voice recognition API. The input is the voice data sent from the device, and the output is the text data converted from the voice. Through this process, the content of the conversation is visualized as text.

[0455] Step 5:

[0456] (emotional analysis)

[0457] The server analyzes the user's emotions using text data and voice tone data. The input is voice data and text data, and the output is the user's emotional state (e.g., joy, anger, sadness). The emotion recognition engine analyzes the voice tone and text content.

[0458] Step 6:

[0459] (Generation of sign language video data)

[0460] The server uses a generative AI model to convert the parsed text data into sign language video data. The input is text data, and the output is basic sign language video data. The generative AI model (e.g., a Transformer model) converts the text into sign language actions.

[0461] Step 7:

[0462] (Emotional modulation of sign language video data)

[0463] The server adjusts the expression of the sign language video data based on the recognized emotion information. The input is basic sign language video data and emotion information, and the output is adjusted sign language video data. The speed and movement of the sign language are changed to reflect the emotion.

[0464] Step 8:

[0465] (Sign language video data encoding)

[0466] The server encodes the sign language video data into an appropriate multimedia format. The input is the adapted sign language video data, and the output is an encoded sign language video file. This process allows the data to be transferred efficiently.

[0467] Step 9:

[0468] (Transmission of sign language video data)

[0469] The server sends encoded sign language video data to the terminal. The input is the encoded sign language video data, and the output is the data to be sent to the terminal. HTTPS is used as the communication protocol, and the data is encrypted.

[0470] Step 10:

[0471] (Receiving and displaying sign language video data)

[0472] The terminal receives the sign language video data sent from the server and displays it on the display device. The input is the sign language video data sent from the server, and the output is a sign language video that the user can visually confirm. Buffering processing is performed to ensure that the video is displayed smoothly.

[0473] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0474] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0475] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0476] [Second embodiment]

[0477] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0478] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0479] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0480] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0481] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0482] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0483] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0484] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0485] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0486] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0487] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0488] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0489] This invention is a system that converts audio data into sign language video data in real time and displays it to solve the shortage of sign language interpreters and the difficulty of allocating them. Specifically, sign language interpretation is realized by acquiring audio data, converting it into text data, converting it into sign language video data using a generative model, and finally displaying the sign language video on a display means.

[0490] Program processing

[0491] Audio collection and transmission

[0492] 1. The user starts speaking

[0493] User: Speaks into the microphone. Used in meetings, lectures, and everyday conversations.

[0494] 2. The device collects the audio data

[0495] Terminal: Collects user voice data in digital format in real time via a microphone.

[0496] 3. The device preprocesses the audio data

[0497] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[0498] 4. The device sends the audio data to the server

[0499] Terminal: Sends the pre-processed audio data to the server using a secure communication protocol.

[0500] Analysis of voice data and sign language conversion

[0501] 5. The server receives the audio data

[0502] Server: Receives the audio data sent from the terminal and establishes a data stream.

[0503] 6. The server converts the audio data into text data

[0504] Server: Using voice recognition technology, the received voice data is converted into text data.

[0505] 7. The server processes the text data using natural language processing

[0506] Server: Analyzes the generated text data using a natural language processing algorithm and converts it into a format suitable for sign language.

[0507] 8. The server generates sign language video using the generative model

[0508] Server: Using a trained generative model, it generates animated sign language videos based on text data.

[0509] Sign language video generation and display

[0510] 9. The server encodes the sign language video data

[0511] Server: Encodes the generated sign language video data into an appropriate multimedia format.

[0512] 10. The server sends the sign language video data to the device.

[0513] Server: Sends encoded sign language video data to the terminal using a secure communication protocol.

[0514] 11. The device receives the sign language video data

[0515] Terminal: Receives sign language video data sent from the server and stores it in the display buffer.

[0516] 12. The device displays sign language images

[0517] Terminal: The received sign language video is displayed on a monitor or smart glasses display, allowing the user to visually understand the sign language interpretation in real time.

[0518] Specific examples

[0519] 1. Use in meetings

[0520] User: The meeting facilitator makes a presentation using a microphone.

[0521] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0522] Server: Receives voice data and converts it into text data using speech recognition technology. Generates sign language video data using a generative model, encodes it, and sends it to the device.

[0523] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[0524] 2. Daily communication

[0525] User: Starts speaking to place an order at a cafe.

[0526] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0527] Server: Converts voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[0528] Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[0529] As described above, the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving access to information for the hearing impaired.

[0530] The processing flow will be explained below.

[0531] Step 1:

[0532] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0533] Step 2:

[0534] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0535] Step 3:

[0536] Terminal: Preprocessing such as noise reduction and volume normalization is performed on the collected audio data. Here, signal processing algorithms are used to adjust the audio data so that it is clear.

[0537] Step 4:

[0538] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[0539] Step 5:

[0540] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[0541] Step 6:

[0542] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0543] Step 7:

[0544] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[0545] Step 8:

[0546] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[0547] Step 9:

[0548] Server: Encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces the file size and improves transmission and display efficiency.

[0549] Step 10:

[0550] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[0551] Step 11:

[0552] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[0553] Step 12:

[0554] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand the smooth sign language interpretation.

[0555] As described above, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving information access for the hearing impaired.

[0556] Example 1

[0557] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] Currently, there is a shortage of sign language interpreters and difficulty in locating them, which poses a major barrier for the hearing impaired, especially in situations where real-time communication is required. In addition, the technology for converting speech information into sign language is still in its infancy, and improvements in accuracy and speed are required. To solve these issues, a system that provides high-quality, real-time sign language interpretation is needed.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0560] In this invention, the server includes a speech recognition unit that converts speech data into text data, a unit that performs natural language processing on the text data and converts it into a format suitable for sign language video data, and a unit that converts the text data into sign language video data using a generative model, thereby enabling high-quality real-time sign language interpretation.

[0561] The "means for the user to produce voice" is a system component that includes the user inputting voice into a microphone during a meeting or everyday conversation.

[0562] The "means for the terminal to collect the voice as digital voice data" refers to the function of converting analog voice into a digital signal and collecting it in real time through a microphone connected to the terminal.

[0563] The "means for preprocessing the audio data and performing noise removal and volume normalization" refers to software or hardware functions for removing unnecessary noise from the audio data and performing processing to maintain a constant volume.

[0564] The "means for transmitting preprocessed audio data to a server" is a device or method for transferring the preprocessed audio data to a server via a secure communications protocol.

[0565] The "server means for receiving the voice data" is a function of the server for receiving and processing the voice data transmitted from the terminal.

[0566] The "voice recognition means for converting the voice data into text data" refers to a technology or software function for analyzing a voice signal and converting the content into text information.

[0567] "Means for natural language processing the text data and converting it into a format suitable for sign language video data" refers to algorithms or software functions for performing natural language processing on text data, extracting and organizing the information necessary for sign language expression.

[0568] "Means for converting the text data into sign language video data using a generative model" refers to technology or software functions that convert text data into sign language animation video using a generative model.

[0569] "Means for encoding the sign language video data into a suitable multimedia format" refers to software or hardware functionality for compressing and encoding the generated sign language video data into a suitable multimedia format.

[0570] The "means for transmitting the encoded sign language video data to the receiving terminal" refers to a communication means or protocol for securely transferring the encoded sign language video data to the terminal.

[0571] The "means for displaying the received sign language video data" refers to a display device or software function for visually displaying the received sign language video data to the user on the terminal.

[0572] The "means for displaying the sign language video data on a display device" refers to hardware and software functions for displaying the sign language video data on a terminal display or an external monitor.

[0573] "Means for using a microphone and audio processing software to collect and pre-process said audio data" refers to the function of using a microphone to collect audio and pre-processing the collected audio with software.

[0574] This invention is a system that converts audio data into sign language video data in real time and displays it in order to solve the shortage of sign language interpreters and the difficulty of allocating them. How this system is implemented will be explained below in detail.

[0575] First, situations where a user may speak include a meeting or everyday conversation. The user inputs the voice through a microphone. For example, a general microphone is used in a meeting.

[0576] Next, the device, such as a laptop, collects this voice in real time as digital audio data. Specifically, a microphone connected to the device converts analog audio into a digital signal, and the collected audio data is stored in the device.

[0577] The collected voice data is pre-processed within the device, which includes noise reduction and volume normalization. The voice processing software used is, for example, general voice processing software.

[0578] The pre-processed audio data is then sent to the server using a secure communication protocol, TLS / SSL, which encrypts the data during transmission.

[0579] The server receives the voice data sent from the device. For example, a general-purpose server used as part of an online service is used as the server. The server uses the received voice data for analysis as is.

[0580] The server then converts the voice data into text data. This voice recognition process uses, for example, voice recognition technology. Specifically, a voice recognition engine analyzes the voice signal and converts it into text data.

[0581] The generated text data is then subjected to natural language processing and converted into a format suitable for sign language video data. A general natural language processing library is often used as the natural language processing algorithm. This analyzes the structure of the sentence and extracts the information necessary for sign language expression.

[0582] The converted text data is then converted into sign language video data using a generative model, such as a trained generative AI model. At this stage, the text data is converted into sign language animation video.

[0583] The resulting sign language video data is then encoded into a suitable multimedia format, for example using multimedia software, which compresses the video data and optimizes it for easy transmission and playback.

[0584] The encoded sign language video data is securely transmitted from the server to a receiving terminal, such as a smartphone carried by a user.

[0585] The receiving terminal receives the sign language video data and displays it on a display device, allowing the user to visually understand the sign language interpretation in real time.

[0586] Specific examples

[0587] Use in meetings

[0588] 1. User: The meeting facilitator makes a presentation using a standard microphone.

[0589] 2. Terminal: A laptop computer installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0590] 3. Server: Receives the voice data and converts it into text data using speech recognition technology. It then generates sign language video data using a generative model, encodes it, and sends it to the device.

[0591] 4. Terminal: The received sign language video is displayed on the screen in the conference room in real time. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[0592] Daily communication

[0593] 1. User: Starts speaking to place an order at a cafe.

[0594] 2. Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0595] 3. Server: Converts the voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[0596] 4. Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[0597] Prompt Sentence Examples

[0598] "I would like to convert what is being said in a meeting into sign language in real time. Please tell me how to use this system."

[0599] "Please explain the setup procedure for displaying orders in sign language at a cafe."

[0600] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0601] Step 1:

[0602] The user speaks.

[0603] During meetings or everyday conversations, users input their voice into a microphone. This voice becomes the input to the system. The voice uttered by the user is sent to the terminal via the microphone.

[0604] Step 2:

[0605] The terminal collects the voice data as digital voice data.

[0606] A microphone connected to the device converts analog voice into a digital signal and collects it in real time. The collected digital voice data is stored in the device's memory. Analog voice data is the input, and digital voice data is obtained as the output.

[0607] Step 3:

[0608] The device preprocesses the audio data.

[0609] The device performs preprocessing on the collected audio data, such as noise reduction and volume normalization. Audio processing software is used to remove noise and maintain a constant volume. The input for preprocessing is digital audio data, and the output is the preprocessed audio data.

[0610] Step 4:

[0611] The terminal transmits the preprocessed audio data to the server.

[0612] The preprocessed audio data is sent to the server using a secure communication protocol (TLS / SSL). At this time, the data is encrypted and sent. The input is the preprocessed audio data, and the output is the data sent to the server.

[0613] Step 5:

[0614] The server receives the audio data.

[0615] The server receives the voice data sent from the terminal. The received voice data is stored in the server's memory. The input is the voice data sent from the terminal, and the output is the received voice data.

[0616] Step 6:

[0617] The server converts the voice data into text data.

[0618] The server uses speech recognition technology to convert the received voice data into text data. The speech recognition engine analyzes the voice signal and generates a corresponding text representation. The input is voice data and the output is text data.

[0619] Step 7:

[0620] The server performs natural language processing on the text data.

[0621] The generated text data is analyzed using natural language processing algorithms and converted into a format suitable for sign language. Sentence structure is analyzed and important information is extracted. The input is text data, and the output is data in a format suitable for sign language video data.

[0622] Step 8:

[0623] The server generates sign language video using the generative model.

[0624] Based on the generated data, a trained generative model is used to generate animated sign language videos. The input is data in a format suitable for sign language video data, and the output is sign language video data.

[0625] Step 9:

[0626] The server encodes the sign language video data.

[0627] The server encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4 format). Multimedia software is used for the encoding process. The input is the sign language video data, and the output is the encoded sign language video data.

[0628] Step 10:

[0629] The server transmits the encoded sign language video data to the terminal.

[0630] The encoded sign language video data is transmitted to a terminal using a secure communication protocol, where the input is the encoded sign language video data and the output is the data transmitted to the terminal.

[0631] Step 11:

[0632] The terminal receives the sign language video data.

[0633] The terminal receives the encoded sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent to the terminal, and the output is the received sign language video data.

[0634] Step 12:

[0635] The device displays the sign language video.

[0636] The terminal displays the received sign language video data on a display device, such as a screen in a conference room or smart glasses. The input is the received sign language video data, and the output is the displayed sign language video.

[0637] (Application example 1)

[0638] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0639] Modern autonomous driving technology faces the problem of hearing-impaired people having difficulty understanding instructions from voice assistants. Furthermore, in situations where real-time sign language interpretation is not available, hearing-impaired people have difficulty obtaining information quickly and accurately. This can lead to reduced safety and convenience while driving. Therefore, there is a need for the development of a system that converts instructions from voice assistants into sign language images in real time and displays them through smart glasses.

[0640] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0641] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a generative model means for converting the text data into sign language video data, thereby enabling instructions from the voice assistant to be converted into sign language video in real time and displayed through the smart glasses.

[0642] "Audio data" is a digital representation of sound collected by an acoustic sensor such as a microphone.

[0643] "Text data" is data in which voice data is expressed as character information.

[0644] A "generative model" is an artificial intelligence model that generates new data based on input data.

[0645] "Sign language video data" is video data that visually expresses actions corresponding to sign language.

[0646] A "display means" is a hardware or software device for visually presenting an image to a user.

[0647] "Smart glasses" are eyeglass-like devices worn by the user that can display information on a screen.

[0648] "Preprocessing" refers to performing quality improvement processes such as noise removal and volume normalization on the collected data.

[0649] A "multimedia format" is a digital format for uniformly handling different types of data, such as audio, video, and text data.

[0650] "Encoding" is the process of converting data into a particular format.

[0651] A "server" is a computer system for processing data.

[0652] The present invention is a system for converting voice data into sign language video data in real time and displaying it. This system is intended for use in autonomous vehicles, and displays voice instructions and navigation information from a driving assistance system as sign language video.

[0653] The system of the present invention includes the following hardware and software.

[0654] Hardware and software used

[0655] Hardware: smart glasses, microphones, in-car computer systems

[0656] Software: speech recognition engines, natural language processing libraries, generative AI models (e.g., Transformers), sign language animation generation engines, communication protocols (e.g., HTTPS)

[0657] Program processing

[0658] Imagine a scenario in which a user wearing smart glasses receives instructions from a voice assistant inside a car.

[0659] First, the user hears voice commands from the voice assistant. This voice data is collected by a microphone installed in the vehicle. The collected voice data undergoes pre-processing such as noise reduction and volume normalization to improve the data quality. The pre-processed voice data is then sent to a server via the vehicle's computer system using a secure communication protocol.

[0660] The voice data received by the server is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then analyzed by a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converted into a format suitable for sign language. After that, a generative AI model (e.g., GPT-3) is used to generate sign language video data based on the analyzed text data. This generated sign language video data is then encoded into an appropriate multimedia format and sent back to the in-vehicle system.

[0661] The vehicle's onboard computer system receives the sign language video data sent from the server and displays it on the smart glasses' display, allowing the user to understand voice instructions in sign language in real time.

[0662] Specific examples

[0663] Example 1: An in-car voice assistant instructs the driver to "turn right at the next traffic light." The microphone collects the audio and sends it to the server after preprocessing. The server converts the audio instruction into text and generates sign language using natural language processing. The sign language video data is displayed on the smart glasses, allowing the user to understand the instruction through sign language.

[0664] Example 2: An in-car voice assistant instructs, "Set the vehicle speed to 50 km / h." Similarly, voice data is collected, pre-processed, transmitted, and converted into text data. The generated sign language video data is displayed on the smart glasses, allowing the user to understand the instruction.

[0665] Example prompts to input to the generative AI model

[0666] "Please convert the text "Turn right at the next traffic light" into video data of sign language."

[0667] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0668] Step 1:

[0669] The device collects voice instructions using the in-car microphone. Specifically, when an instruction is given from the voice assistant, the microphone converts the analog voice data into a digital format and captures it as a data stream. The input of this step is an analog voice signal, and the output is digital voice data.

[0670] Step 2:

[0671] The digital audio data collected by the device is preprocessed, such as by noise removal and volume normalization. Specifically, an audio processing library (e.g., FFmpeg) is used to improve the quality of the collected data. The input of this step is the digital audio data, and the output is the preprocessed audio data.

[0672] Step 3:

[0673] The terminal transmits the preprocessed voice data to the server via the in-vehicle system using a secure communication protocol (e.g., HTTPS). Specifically, the encoded data is encrypted and transmitted using the HTTPS protocol. The input of this step is the preprocessed voice data, and the output is the data transmitted to the server.

[0674] Step 4:

[0675] The server converts the voice data received from the device into text data using a voice recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice recognition algorithm analyzes the voice characteristics and converts them into a corresponding string of characters. The input of this step is the voice data received by the server, and the output is text data.

[0676] Step 5:

[0677] The server analyzes the generated text data using a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converts it into a format suitable for sign language. Specifically, the text data is semantically analyzed and broken down into phrases that correspond to sign language. The input to this step is text data, and the output is text data in a format suitable for sign language.

[0678] Step 6:

[0679] The server uses a generative AI model (e.g., GPT-3) to generate sign language video data based on the analyzed text data. Specifically, the generative AI model generates appropriate sign language movements and outputs the data as 3D animation. The input for this step is text data in a format suitable for sign language, and the output is sign language video data.

[0680] Step 7:

[0681] The server encodes the generated sign language video data into an appropriate multimedia format. Specifically, it encodes it into a video format (e.g., MP4) and prepares it for transmission. The input of this step is the sign language video data, and the output is the encoded sign language video data.

[0682] Step 8:

[0683] The server then sends the encoded sign language video data to the terminal again using a secure communication protocol. Specifically, the encoded data is encrypted and sent using the HTTPS protocol. The input to this step is the encoded sign language video data, and the output is the data sent to the terminal.

[0684] Step 9:

[0685] The terminal receives the sign language video data sent from the server and displays it on the smart glasses display. Specifically, it decodes the received data and visually displays it to the user. The input of this step is the sign language video data sent from the server, and the output is the sign language video displayed on the smart glasses display.

[0686] Step 10:

[0687] The user visually understands the sign language video displayed through the smart glasses. Specifically, the user looks at the smart glasses display and understands the instructions from the voice assistant through the sign language video. The input for this step is the sign language video displayed on the smart glasses display, and the output is the user's understanding.

[0688] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0689] This system aims to improve information access for the hearing impaired by providing high-quality real-time sign language interpretation even when a sign language interpreter is not available. In particular, by combining it with an emotion engine that recognizes the user's emotions, it realizes sign language interpretation that responds to the user's emotions.

[0690] Program processing

[0691] Audio collection and transmission

[0692] 1. The user starts speaking

[0693] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0694] 2. The device collects the audio data

[0695] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0696] 3. The device preprocesses the audio data

[0697] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[0698] 4. The device sends the audio data to the server

[0699] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[0700] Analysis of voice data and sign language conversion

[0701] 5. The server receives the audio data

[0702] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[0703] 6. The server converts the audio data into text data

[0704] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0705] 7. The server processes the text data using natural language processing

[0706] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[0707] 8. The server generates sign language video using the generative model

[0708] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[0709] 9. The server recognizes the user's emotions using an emotion engine

[0710] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[0711] 10. The server adjusts the sign language video data based on emotions.

[0712] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, if the user expresses anger, the server may adjust the expression by emphasizing the sign language movements.

[0713] 11. The server encodes the sign language video data

[0714] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[0715] 12. The server sends the sign language video data to the device.

[0716] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[0717] 13. The device receives sign language video data

[0718] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[0719] 14. The device displays sign language images

[0720] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[0721] Specific examples

[0722] 1. Use in meetings

[0723] User: The meeting facilitator makes a presentation using a microphone.

[0724] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0725] Server: Receives voice data and converts it into text using speech recognition technology. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the sign language video. After encoding, it is sent to the device.

[0726] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video to understand the content of the presentation and also grasp the presenter's emotions.

[0727] 2. Daily communication

[0728] User: Starts speaking to place an order at a cafe.

[0729] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0730] Server: Converts voice data into text using speech recognition technology and analyzes it using natural language processing algorithms. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the video. After encoding, it is sent to the device.

[0731] Device: The received sign language video is displayed on the smart glasses. Through the video, the hearing-impaired person can understand the order details and also grasp the emotions of the store clerk.

[0732] As described above, the system of the present invention provides high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired and understanding the user's emotions.

[0733] The processing flow will be explained below.

[0734] Step 1:

[0735] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0736] Step 2:

[0737] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0738] Step 3:

[0739] Terminal: The collected audio data is preprocessed using noise reduction and volume normalization to improve the quality of the data. Specifically, the audio signal is processed using a digital filter to reduce background noise.

[0740] Step 4:

[0741] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with Transport Layer Security (TLS).

[0742] Step 5:

[0743] Server: Receives the audio data sent from the device, establishes a data stream, and temporarily stores the received data in local storage.

[0744] Step 6:

[0745] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0746] Step 7:

[0747] Server: The generated text data is analyzed using a natural language processing (NLP) algorithm and converted into a format suitable for sign language. The NLP algorithm performs grammatical and semantic analysis to generate a text format that is optimal for sign language conversion.

[0748] Step 8:

[0749] Server: Converts the analyzed text data into animated sign language videos using a trained generative model. The generative model (e.g., a Transformer model) includes a neural network that inputs text data and generates sign language videos.

[0750] Step 9:

[0751] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[0752] Step 10:

[0753] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, it emphasizes the sign language movements to indicate anger, and softens the sign language movements to indicate joy.

[0754] Step 11:

[0755] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[0756] Step 12:

[0757] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with TLS.

[0758] Step 13:

[0759] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[0760] Step 14:

[0761] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[0762] As a result, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired. Furthermore, by understanding the user's emotions and providing appropriate sign language expressions, the system achieves more natural and empathetic communication.

[0763] Example 2

[0764] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0765] It is difficult for hearing-impaired people to obtain accurate and emotionally rich information in real time even in situations where a sign language interpreter is not available. Furthermore, there are an increasing number of situations where communication that includes emotional nuances, rather than just conveying information, is required. The purpose of this invention is to solve these problems and improve information access for hearing-impaired people.

[0766] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0767] In this invention, the server includes means for acquiring audio data, means for preprocessing the audio data, means for converting the audio data into text data, means for generative modeling for converting the text data into sign language video data, means for adjusting the sign language video data based on a user's emotions, means for encoding the sign language video data into an appropriate multimedia format, and means for displaying the sign language video data, thereby enabling real-time, high-quality sign language interpretation and the generation of sign language video that reflects the user's emotions.

[0768] The "means for acquiring voice data" refers to a device or system that collects voices uttered by a user as digital data.

[0769] "Means for preprocessing audio data" refers to a device or system that performs processes such as noise removal and volume normalization on collected audio data to improve the quality of the data.

[0770] "Means for converting voice data into text data" refers to a device or system that uses voice recognition technology to convert voice data into text information.

[0771] "Generative model means for converting text data into sign language video data" refers to a device or system that uses a generative model algorithm to convert text information into sign language animation video.

[0772] The "means for adjusting sign language video data based on the user's emotions" refers to a device or system that analyzes the user's emotional state and adjusts the movement of the sign language video accordingly.

[0773] A "means for encoding sign language video data into a suitable multimedia format" is a device or system that converts sign language video data into a suitable multimedia format (e.g., MP4) for more efficient transmission and display.

[0774] "Means for displaying sign language video data" refers to a device or system that displays the generated and adjusted sign language video data on a device such as a monitor or smart glasses.

[0775] A "means for transmitting audio data using a secure communication protocol" is a device or system that transmits captured and pre-processed audio data to a server in a secure and reliable manner (e.g., HTTPS).

[0776] "Means for recognizing user emotions from sign language video data using an emotion engine" refers to a set of algorithms and devices for analyzing and recognizing user emotions from audio data and text data.

[0777] MODE FOR CARRYING OUT THE INVENTION

[0778] This invention provides a system that provides high-quality real-time sign language interpretation even when a sign language interpreter is not available, improving information access for the hearing impaired. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to realize sign language interpretation that responds to emotions.

[0779] Hardware and Software

[0780] Hardware:

[0781] 1. Device:

[0782] High-sensitivity microphones: used to collect audio precisely. An example is the Shure SM58.

[0783] Display: Use a monitor or smart glasses (e.g., Google Glass) to visually display the sign language image.

[0784] 2. Server:

[0785] Use servers with high-performance computing resources, such as cloud-based servers (e.g., AWS, Google Cloud).

[0786] software:

[0787] 1. Device:

[0788] Audio pre-processing software: Use the noise reduction feature in Audacity.

[0789] Communication Protocol: HTTPS and TLS are used to ensure secure communication.

[0790] 2. Server:

[0791] Speech recognition technology: Google Cloud Speech-to-Text API.

[0792] Natural language processing algorithms: SpaCy, NLTK.

[0793] Generative model: Generates sign language animation video using a Transformers model.

[0794] Sentiment Engine: IBM Watson sentiment analysis API.

[0795] Encoding tool: Use FFmpeg to encode the sign language video data into MP4 format.

[0796] Overview of program processing

[0797] 1. User:

[0798] Speak into the microphone. The system can be used in a variety of situations, such as meetings and everyday conversations.

[0799] 2. Terminal:

[0800] Collect user voice data in real time through a microphone.

[0801] Preprocess the audio data using Audacity's noise reduction function.

[0802] The preprocessed audio data is encrypted with TLS using HTTPS and sent to the server.

[0803] 3. Server:

[0804] A data stream is established to receive and temporarily store audio data sent from the terminal.

[0805] Convert the audio data into text data using the Google Cloud Speech-to-Text API.

[0806] Using SpaCy and NLTK, text data is processed using natural language processing and converted into a format suitable for sign language.

[0807] Generate animated sign language videos using Transformers models.

[0808] Using IBM Watson's sentiment analysis API, it recognizes user emotions from voice and text data.

[0809] Adjust sign language video data based on recognized emotions, for example, enhancing sign movements to indicate anger.

[0810] Use an encoding tool (FFmpeg) to encode the adjusted sign language video data into MP4 format.

[0811] The encoded sign language video data is sent to the terminal using HTTPS.

[0812] 4. Terminal:

[0813] The sign language video data transmitted from the server is received and stored in the display buffer.

[0814] The received sign language video is displayed on a monitor or smart glasses display.

[0815] Specific examples

[0816] Meeting use:

[0817] A user acts as the meeting facilitator and makes a presentation using a microphone. A device (e.g., a laptop PC) installed in the conference room collects audio in real time, preprocesses it, and sends it to a server. The server converts the received audio data into text using the Google Cloud Speech-to-Text API, and generates a sign language video using a generative model and emotion engine. The video data is encoded, sent back to the device, and displayed on a screen in the conference room. Hearing-impaired participants can understand the content of the speech and the speaker's emotions through this sign language video.

[0818] Daily communication:

[0819] A user begins speaking to place an order at a cafe. The smartphone collects the audio, preprocesses it using Audacity, and sends the audio data to a server. The server converts the audio data into text using the Google Cloud Speech-to-Text API and creates a sign language video using a generative model and emotion engine. The adjusted and encoded data is sent back to the device and displayed on the smart glasses display. The hearing-impaired user can understand the order and the waiter's emotions through this video.

[0820] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0821] Step 1:

[0822] The user speaks

[0823] A user speaks into a microphone for meetings or everyday conversations. The input is the user's spoken voice. The output is an analog audio signal.

[0824] Step 2:

[0825] The device collects the audio

[0826] The device uses a high-sensitivity microphone (e.g., Shure SM58) to collect the user's voice as a digital signal. The input is an analog audio signal. The output is digital audio data.

[0827] Step 3:

[0828] The device preprocesses the audio

[0829] The device uses Audacity's noise reduction function to remove noise from the collected audio data and normalizes the volume. The input is digital audio data. The output is pre-processed, clear audio data.

[0830] Step 4:

[0831] The device sends the audio data to the server.

[0832] The device sends the preprocessed audio data to the server using HTTPS, and the TLS protocol is used for data encryption. The input is the preprocessed audio data. The output is the audio data sent to the server via a secure communication protocol.

[0833] Step 5:

[0834] The server receives the audio data.

[0835] The server receives the voice data sent from the terminal and stores it in a temporary data stream, which is used for subsequent processing. The input is the voice data sent from the terminal. The output is the temporarily stored voice data.

[0836] Step 6:

[0837] The server converts the audio data into text

[0838] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data. The output is a highly accurate text data.

[0839] Step 7:

[0840] The server processes the text in natural language.

[0841] The server analyzes the generated text data using natural language processing algorithms such as SpaCy and NLTK and converts it into a format suitable for sign language. The input is text data. The output is data in a format suitable for sign language.

[0842] Step 8:

[0843] The server generates the sign language video

[0844] The server uses a Transformers model to convert natural language processed text data into sign language animation. The input is data in a format suitable for sign language. The output is sign language animation data.

[0845] Step 9:

[0846] The server recognizes emotions

[0847] The server uses IBM Watson's sentiment analysis API to recognize the user's emotions from voice and text data. The input is voice and text data. The output is the recognized emotion information.

[0848] Step 10:

[0849] The server adjusts the sign language video based on emotions.

[0850] The server adjusts the sign language video data based on the recognized emotion. For example, if the emotion is "sadness," the sign language movements are slowed down. The input is the sign language video data and the emotion information. The output is the adjusted sign language video data.

[0851] Step 11:

[0852] The server encodes the sign language video

[0853] The server encodes the adjusted sign language video data into MP4 format using FFmpeg. The input is the adjusted sign language video data. The output is the encoded sign language video data in MP4 format.

[0854] Step 12:

[0855] The server sends the sign language video data

[0856] The server sends the encoded sign language video data to the terminal using HTTPS to ensure reliability and security of the communication. The input is the encoded sign language video data. The output is the sign language video data sent to the terminal.

[0857] Step 13:

[0858] The device receives the sign language video data.

[0859] The terminal receives sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent from the server. The output is the sign language video data stored in the display buffer.

[0860] Step 14:

[0861] The device displays a sign language video.

[0862] The device displays the received sign language video on a monitor or smart glasses (e.g., Google Glass). The input is the sign language video data stored in the display buffer. The output is the sign language video displayed on the monitor or smart glasses.

[0863] (Application example 2)

[0864] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0865] Current sign language interpretation systems have difficulty reflecting the user's emotions appropriately, making it difficult to convey emotional nuances. Furthermore, it is difficult to provide high-quality real-time sign language interpretation when a sign language interpreter is unavailable. This makes it difficult for the hearing impaired to accurately understand information.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0867] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for generating a generative model for converting the text data into sign language video data, means for analyzing a user's emotions, means for adjusting the sign language video data based on the user's emotions, and means for displaying the sign language video data. This enables high-quality real-time sign language interpretation that reflects the user's emotions even in places where a sign language interpreter is not available, enabling people with hearing impairments to accurately understand information.

[0868] The "means for acquiring voice data" refers to a means for collecting the voice uttered by the user in digital form.

[0869] The "means for converting voice data into text data" refers to a means for converting collected voice data into text information using natural language processing technology.

[0870] The "generative model means for converting text data into sign language video data" is a means for simulating sign language movements based on character information and generating the results as video data.

[0871] The "means for displaying sign language video data" refers to a means for displaying the generated sign language video on a display device such as a monitor or smart glasses.

[0872] "Means for analyzing user emotions" refers to means for reading and analyzing the user's emotional state from voice data or text data.

[0873] The "means for adjusting sign language video data based on the user's emotions" refers to a means for adjusting the movements and expressions of the sign language video to reflect the analyzed emotions of the user.

[0874] The present invention is a system that acquires voice data, analyzes the voice data, and converts it into sign language video data in real time. Furthermore, it can realize natural and emotional communication by identifying the user's emotions and adjusting the sign language video according to those emotions. The following hardware and software are used to implement the present invention.

[0875] Hardware used

[0876] 1. Audio collection device: Use smart glasses or a smartphone equipped with a highly sensitive microphone.

[0877] 2. Display device: Use smart glasses or a monitor equipped with a display for displaying sign language images.

[0878] 3. Data processing server: A high-performance server is used to process and manage audio data and sign language video data.

[0879] Software used

[0880] 1. Speech recognition software: Uses a speech recognition API to convert voice data into text data.

[0881] 2. Natural Language Processing Algorithms: Use natural language processing (NLP) algorithms to analyze text data.

[0882] 3. Emotion Recognition Engine: An emotion recognition engine is used to analyze user emotions from voice and text.

[0883] 4. Sign language generative model: Use a generative AI model (e.g., Transformer model) that converts text into sign language.

[0884] 5. Video Encoder: Use multimedia software to encode the generated sign language video data into an appropriate format.

[0885] System Functions and Flow

[0886] 1. Audio collection: The microphone installed on the device is used to acquire the user's audio data.

[0887] 2. Speech recognition: The collected voice data is sent to the server and converted into text data using a speech recognition API.

[0888] 3. Sentiment analysis: Recognize user emotions from text data and original audio data.

[0889] 4. Sign Language Generation: Use a generative AI model to convert text data into sign language video data.

[0890] 5. Emotional adjustment: Adjust the expression of sign language video based on the recognized emotional information.

[0891] 6. Video display: After buffering, the adjusted sign language video data is displayed on the display device.

[0892] Specific examples

[0893] In a physical store, when a cashier asks a customer, "Do you have a point card?", this system allows the cashier to display a sign language image through the smart glasses. Furthermore, if the emotion of "happiness" is analyzed from the customer's attitude and facial expression, the sign language image is adjusted to reflect that emotion, resulting in more natural and human-like communication.

[0894] Prompt Sentence Examples

[0895] "Real-time sign language interpretation when a cashier asks, 'Do you have a loyalty card?'"

[0896] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0897] Step 1:

[0898] (Audio collection)

[0899] The device captures the user's voice in real time through a microphone. The input is the user's speech, and the output is digital voice data. The device uses a highly sensitive microphone to improve the accuracy of the collected voice data.

[0900] Step 2:

[0901] (Preprocessing of audio data)

[0902] The device performs preprocessing on the collected voice data, such as noise removal and volume normalization, to improve the quality of the data. The input is the raw voice data acquired in step 1, and the output is clear, preprocessed voice data. This processing improves the accuracy of voice recognition.

[0903] Step 3:

[0904] (Sending audio data)

[0905] The device sends the preprocessed audio data to the server. The input is the preprocessed audio data, and the output is a secure data transfer to the server. HTTPS is used as the communication protocol, and the data is transmitted encrypted.

[0906] Step 4:

[0907] (Speech recognition of voice data)

[0908] The server converts the received voice data into text data using a voice recognition API. The input is the voice data sent from the device, and the output is the text data converted from the voice. Through this process, the content of the conversation is visualized as text.

[0909] Step 5:

[0910] (emotional analysis)

[0911] The server analyzes the user's emotions using text data and voice tone data. The input is voice data and text data, and the output is the user's emotional state (e.g., joy, anger, sadness). The emotion recognition engine analyzes the voice tone and text content.

[0912] Step 6:

[0913] (Generation of sign language video data)

[0914] The server uses a generative AI model to convert the parsed text data into sign language video data. The input is text data, and the output is basic sign language video data. The generative AI model (e.g., a Transformer model) converts the text into sign language actions.

[0915] Step 7:

[0916] (Emotional modulation of sign language video data)

[0917] The server adjusts the expression of the sign language video data based on the recognized emotion information. The input is basic sign language video data and emotion information, and the output is adjusted sign language video data. The speed and movement of the sign language are changed to reflect the emotion.

[0918] Step 8:

[0919] (Sign language video data encoding)

[0920] The server encodes the sign language video data into an appropriate multimedia format. The input is the adapted sign language video data, and the output is an encoded sign language video file. This process allows the data to be transferred efficiently.

[0921] Step 9:

[0922] (Transmission of sign language video data)

[0923] The server sends encoded sign language video data to the terminal. The input is the encoded sign language video data, and the output is the data to be sent to the terminal. HTTPS is used as the communication protocol, and the data is encrypted.

[0924] Step 10:

[0925] (Receiving and displaying sign language video data)

[0926] The terminal receives the sign language video data sent from the server and displays it on the display device. The input is the sign language video data sent from the server, and the output is a sign language video that the user can visually confirm. Buffering processing is performed to ensure that the video is displayed smoothly.

[0927] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0928] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0929] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0930] [Third embodiment]

[0931] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0932] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0933] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0934] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0935] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0936] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0937] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0938] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0939] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0940] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0941] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0942] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0943] This invention is a system that converts audio data into sign language video data in real time and displays it to solve the shortage of sign language interpreters and the difficulty of allocating them. Specifically, sign language interpretation is realized by acquiring audio data, converting it into text data, converting it into sign language video data using a generative model, and finally displaying the sign language video on a display means.

[0944] Program processing

[0945] Audio collection and transmission

[0946] 1. The user starts speaking

[0947] User: Speaks into the microphone. Used in meetings, lectures, and everyday conversations.

[0948] 2. The device collects the audio data

[0949] Terminal: Collects user voice data in digital format in real time via a microphone.

[0950] 3. The device preprocesses the audio data

[0951] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[0952] 4. The device sends the audio data to the server

[0953] Terminal: Sends the pre-processed audio data to the server using a secure communication protocol.

[0954] Analysis of voice data and sign language conversion

[0955] 5. The server receives the audio data

[0956] Server: Receives the audio data sent from the terminal and establishes a data stream.

[0957] 6. The server converts the audio data into text data

[0958] Server: Using voice recognition technology, the received voice data is converted into text data.

[0959] 7. The server processes the text data using natural language processing

[0960] Server: Analyzes the generated text data using a natural language processing algorithm and converts it into a format suitable for sign language.

[0961] 8. The server generates sign language video using the generative model

[0962] Server: Using a trained generative model, it generates animated sign language videos based on text data.

[0963] Sign language video generation and display

[0964] 9. The server encodes the sign language video data

[0965] Server: Encodes the generated sign language video data into an appropriate multimedia format.

[0966] 10. The server sends the sign language video data to the device.

[0967] Server: Sends encoded sign language video data to the terminal using a secure communication protocol.

[0968] 11. The device receives the sign language video data

[0969] Terminal: Receives sign language video data sent from the server and stores it in the display buffer.

[0970] 12. The device displays sign language images

[0971] Terminal: The received sign language video is displayed on a monitor or smart glasses display, allowing the user to visually understand the sign language interpretation in real time.

[0972] Specific examples

[0973] 1. Use in meetings

[0974] User: The meeting facilitator makes a presentation using a microphone.

[0975] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[0976] Server: Receives voice data and converts it into text data using speech recognition technology. Generates sign language video data using a generative model, encodes it, and sends it to the device.

[0977] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[0978] 2. Daily communication

[0979] User: Starts speaking to place an order at a cafe.

[0980] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[0981] Server: Converts voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[0982] Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[0983] As described above, the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving access to information for the hearing impaired.

[0984] The processing flow will be explained below.

[0985] Step 1:

[0986] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[0987] Step 2:

[0988] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[0989] Step 3:

[0990] Terminal: Preprocessing such as noise reduction and volume normalization is performed on the collected audio data. Here, signal processing algorithms are used to adjust the audio data so that it is clear.

[0991] Step 4:

[0992] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[0993] Step 5:

[0994] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[0995] Step 6:

[0996] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[0997] Step 7:

[0998] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[0999] Step 8:

[1000] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[1001] Step 9:

[1002] Server: Encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces the file size and improves transmission and display efficiency.

[1003] Step 10:

[1004] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[1005] Step 11:

[1006] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[1007] Step 12:

[1008] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand the smooth sign language interpretation.

[1009] As described above, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving information access for the hearing impaired.

[1010] Example 1

[1011] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] Currently, there is a shortage of sign language interpreters and difficulty in locating them, which poses a major barrier for the hearing impaired, especially in situations where real-time communication is required. In addition, the technology for converting speech information into sign language is still in its infancy, and improvements in accuracy and speed are required. To solve these issues, a system that provides high-quality, real-time sign language interpretation is needed.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1014] In this invention, the server includes a speech recognition unit that converts speech data into text data, a unit that performs natural language processing on the text data and converts it into a format suitable for sign language video data, and a unit that converts the text data into sign language video data using a generative model, thereby enabling high-quality real-time sign language interpretation.

[1015] The "means for the user to produce voice" is a system component that includes the user inputting voice into a microphone during a meeting or everyday conversation.

[1016] The "means for the terminal to collect the voice as digital voice data" refers to the function of converting analog voice into a digital signal and collecting it in real time through a microphone connected to the terminal.

[1017] The "means for preprocessing the audio data and performing noise removal and volume normalization" refers to software or hardware functions for removing unnecessary noise from the audio data and performing processing to maintain a constant volume.

[1018] The "means for transmitting preprocessed audio data to a server" is a device or method for transferring the preprocessed audio data to a server via a secure communications protocol.

[1019] The "server means for receiving the voice data" is a function of the server for receiving and processing the voice data transmitted from the terminal.

[1020] The "voice recognition means for converting the voice data into text data" refers to a technology or software function for analyzing a voice signal and converting the content into text information.

[1021] "Means for natural language processing the text data and converting it into a format suitable for sign language video data" refers to algorithms or software functions for performing natural language processing on text data, extracting and organizing the information necessary for sign language expression.

[1022] "Means for converting the text data into sign language video data using a generative model" refers to technology or software functions that convert text data into sign language animation video using a generative model.

[1023] "Means for encoding the sign language video data into a suitable multimedia format" refers to software or hardware functionality for compressing and encoding the generated sign language video data into a suitable multimedia format.

[1024] The "means for transmitting the encoded sign language video data to the receiving terminal" refers to a communication means or protocol for securely transferring the encoded sign language video data to the terminal.

[1025] The "means for displaying the received sign language video data" refers to a display device or software function for visually displaying the received sign language video data to the user on the terminal.

[1026] The "means for displaying the sign language video data on a display device" refers to hardware and software functions for displaying the sign language video data on a terminal display or an external monitor.

[1027] "Means for using a microphone and audio processing software to collect and pre-process said audio data" refers to the function of using a microphone to collect audio and pre-processing the collected audio with software.

[1028] This invention is a system that converts audio data into sign language video data in real time and displays it in order to solve the shortage of sign language interpreters and the difficulty of allocating them. How this system is implemented will be explained below in detail.

[1029] First, situations where a user may speak include a meeting or everyday conversation. The user inputs the voice through a microphone. For example, a general microphone is used in a meeting.

[1030] Next, the device, such as a laptop, collects this voice in real time as digital audio data. Specifically, a microphone connected to the device converts analog audio into a digital signal, and the collected audio data is stored in the device.

[1031] The collected voice data is pre-processed within the device, which includes noise reduction and volume normalization. The voice processing software used is, for example, general voice processing software.

[1032] The pre-processed audio data is then sent to the server using a secure communication protocol, TLS / SSL, which encrypts the data during transmission.

[1033] The server receives the voice data sent from the device. For example, a general-purpose server used as part of an online service is used as the server. The server uses the received voice data for analysis as is.

[1034] The server then converts the voice data into text data. This voice recognition process uses, for example, voice recognition technology. Specifically, a voice recognition engine analyzes the voice signal and converts it into text data.

[1035] The generated text data is then subjected to natural language processing and converted into a format suitable for sign language video data. A general natural language processing library is often used as the natural language processing algorithm. This analyzes the structure of the sentence and extracts the information necessary for sign language expression.

[1036] The converted text data is then converted into sign language video data using a generative model, such as a trained generative AI model. At this stage, the text data is converted into sign language animation video.

[1037] The resulting sign language video data is then encoded into a suitable multimedia format, for example using multimedia software, which compresses the video data and optimizes it for easy transmission and playback.

[1038] The encoded sign language video data is securely transmitted from the server to a receiving terminal, such as a smartphone carried by a user.

[1039] The receiving terminal receives the sign language video data and displays it on a display device, allowing the user to visually understand the sign language interpretation in real time.

[1040] Specific examples

[1041] Use in meetings

[1042] 1. User: The meeting facilitator makes a presentation using a standard microphone.

[1043] 2. Terminal: A laptop computer installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[1044] 3. Server: Receives the voice data and converts it into text data using speech recognition technology. It then generates sign language video data using a generative model, encodes it, and sends it to the device.

[1045] 4. Terminal: The received sign language video is displayed on the screen in the conference room in real time. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[1046] Daily communication

[1047] 1. User: Starts speaking to place an order at a cafe.

[1048] 2. Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[1049] 3. Server: Converts the voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[1050] 4. Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[1051] Prompt Sentence Examples

[1052] "I would like to convert what is being said in a meeting into sign language in real time. Please tell me how to use this system."

[1053] "Please explain the setup procedure for displaying orders in sign language at a cafe."

[1054] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1055] Step 1:

[1056] The user speaks.

[1057] During meetings or everyday conversations, users input their voice into a microphone. This voice becomes the input to the system. The voice uttered by the user is sent to the terminal via the microphone.

[1058] Step 2:

[1059] The terminal collects the voice data as digital voice data.

[1060] A microphone connected to the device converts analog voice into a digital signal and collects it in real time. The collected digital voice data is stored in the device's memory. Analog voice data is the input, and digital voice data is obtained as the output.

[1061] Step 3:

[1062] The device preprocesses the audio data.

[1063] The device performs preprocessing on the collected audio data, such as noise reduction and volume normalization. Audio processing software is used to remove noise and maintain a constant volume. The input for preprocessing is digital audio data, and the output is the preprocessed audio data.

[1064] Step 4:

[1065] The terminal transmits the preprocessed audio data to the server.

[1066] The preprocessed audio data is sent to the server using a secure communication protocol (TLS / SSL). At this time, the data is encrypted and sent. The input is the preprocessed audio data, and the output is the data sent to the server.

[1067] Step 5:

[1068] The server receives the audio data.

[1069] The server receives the voice data sent from the terminal. The received voice data is stored in the server's memory. The input is the voice data sent from the terminal, and the output is the received voice data.

[1070] Step 6:

[1071] The server converts the voice data into text data.

[1072] The server uses speech recognition technology to convert the received voice data into text data. The speech recognition engine analyzes the voice signal and generates a corresponding text representation. The input is voice data and the output is text data.

[1073] Step 7:

[1074] The server performs natural language processing on the text data.

[1075] The generated text data is analyzed using natural language processing algorithms and converted into a format suitable for sign language. Sentence structure is analyzed and important information is extracted. The input is text data, and the output is data in a format suitable for sign language video data.

[1076] Step 8:

[1077] The server generates sign language video using the generative model.

[1078] Based on the generated data, a trained generative model is used to generate animated sign language videos. The input is data in a format suitable for sign language video data, and the output is sign language video data.

[1079] Step 9:

[1080] The server encodes the sign language video data.

[1081] The server encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4 format). Multimedia software is used for the encoding process. The input is the sign language video data, and the output is the encoded sign language video data.

[1082] Step 10:

[1083] The server transmits the encoded sign language video data to the terminal.

[1084] The encoded sign language video data is transmitted to a terminal using a secure communication protocol, where the input is the encoded sign language video data and the output is the data transmitted to the terminal.

[1085] Step 11:

[1086] The terminal receives the sign language video data.

[1087] The terminal receives the encoded sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent to the terminal, and the output is the received sign language video data.

[1088] Step 12:

[1089] The device displays the sign language video.

[1090] The terminal displays the received sign language video data on a display device, such as a screen in a conference room or smart glasses. The input is the received sign language video data, and the output is the displayed sign language video.

[1091] (Application example 1)

[1092] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1093] Modern autonomous driving technology faces the problem of hearing-impaired people having difficulty understanding instructions from voice assistants. Furthermore, in situations where real-time sign language interpretation is not available, hearing-impaired people have difficulty obtaining information quickly and accurately. This can lead to reduced safety and convenience while driving. Therefore, there is a need for the development of a system that converts instructions from voice assistants into sign language images in real time and displays them through smart glasses.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1095] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a generative model means for converting the text data into sign language video data, thereby enabling instructions from the voice assistant to be converted into sign language video in real time and displayed through the smart glasses.

[1096] "Audio data" is a digital representation of sound collected by an acoustic sensor such as a microphone.

[1097] "Text data" is data in which voice data is expressed as character information.

[1098] A "generative model" is an artificial intelligence model that generates new data based on input data.

[1099] "Sign language video data" is video data that visually expresses actions corresponding to sign language.

[1100] A "display means" is a hardware or software device for visually presenting an image to a user.

[1101] "Smart glasses" are eyeglass-like devices worn by the user that can display information on a screen.

[1102] "Preprocessing" refers to performing quality improvement processes such as noise removal and volume normalization on the collected data.

[1103] A "multimedia format" is a digital format for uniformly handling different types of data, such as audio, video, and text data.

[1104] "Encoding" is the process of converting data into a particular format.

[1105] A "server" is a computer system for processing data.

[1106] The present invention is a system for converting voice data into sign language video data in real time and displaying it. This system is intended for use in autonomous vehicles, and displays voice instructions and navigation information from a driving assistance system as sign language video.

[1107] The system of the present invention includes the following hardware and software.

[1108] Hardware and software used

[1109] Hardware: smart glasses, microphones, in-car computer systems

[1110] Software: speech recognition engines, natural language processing libraries, generative AI models (e.g., Transformers), sign language animation generation engines, communication protocols (e.g., HTTPS)

[1111] Program processing

[1112] Imagine a scenario in which a user wearing smart glasses receives instructions from a voice assistant inside a car.

[1113] First, the user hears voice commands from the voice assistant. This voice data is collected by a microphone installed in the vehicle. The collected voice data undergoes pre-processing such as noise reduction and volume normalization to improve the data quality. The pre-processed voice data is then sent to a server via the vehicle's computer system using a secure communication protocol.

[1114] The voice data received by the server is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then analyzed by a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converted into a format suitable for sign language. After that, a generative AI model (e.g., GPT-3) is used to generate sign language video data based on the analyzed text data. This generated sign language video data is then encoded into an appropriate multimedia format and sent back to the in-vehicle system.

[1115] The vehicle's onboard computer system receives the sign language video data sent from the server and displays it on the smart glasses' display, allowing the user to understand voice instructions in sign language in real time.

[1116] Specific examples

[1117] Example 1: An in-car voice assistant instructs the driver to "turn right at the next traffic light." The microphone collects the audio and sends it to the server after preprocessing. The server converts the audio instruction into text and generates sign language using natural language processing. The sign language video data is displayed on the smart glasses, allowing the user to understand the instruction through sign language.

[1118] Example 2: An in-car voice assistant instructs, "Set the vehicle speed to 50 km / h." Similarly, voice data is collected, pre-processed, transmitted, and converted into text data. The generated sign language video data is displayed on the smart glasses, allowing the user to understand the instruction.

[1119] Example prompts to input to the generative AI model

[1120] "Please convert the text "Turn right at the next traffic light" into video data of sign language."

[1121] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1122] Step 1:

[1123] The device collects voice instructions using the in-car microphone. Specifically, when an instruction is given from the voice assistant, the microphone converts the analog voice data into a digital format and captures it as a data stream. The input of this step is an analog voice signal, and the output is digital voice data.

[1124] Step 2:

[1125] The digital audio data collected by the device is preprocessed, such as by noise removal and volume normalization. Specifically, an audio processing library (e.g., FFmpeg) is used to improve the quality of the collected data. The input of this step is the digital audio data, and the output is the preprocessed audio data.

[1126] Step 3:

[1127] The terminal transmits the preprocessed voice data to the server via the in-vehicle system using a secure communication protocol (e.g., HTTPS). Specifically, the encoded data is encrypted and transmitted using the HTTPS protocol. The input of this step is the preprocessed voice data, and the output is the data transmitted to the server.

[1128] Step 4:

[1129] The server converts the voice data received from the device into text data using a voice recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice recognition algorithm analyzes the voice characteristics and converts them into a corresponding string of characters. The input of this step is the voice data received by the server, and the output is text data.

[1130] Step 5:

[1131] The server analyzes the generated text data using a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converts it into a format suitable for sign language. Specifically, the text data is semantically analyzed and broken down into phrases that correspond to sign language. The input to this step is text data, and the output is text data in a format suitable for sign language.

[1132] Step 6:

[1133] The server uses a generative AI model (e.g., GPT-3) to generate sign language video data based on the analyzed text data. Specifically, the generative AI model generates appropriate sign language movements and outputs the data as 3D animation. The input for this step is text data in a format suitable for sign language, and the output is sign language video data.

[1134] Step 7:

[1135] The server encodes the generated sign language video data into an appropriate multimedia format. Specifically, it encodes it into a video format (e.g., MP4) and prepares it for transmission. The input of this step is the sign language video data, and the output is the encoded sign language video data.

[1136] Step 8:

[1137] The server then sends the encoded sign language video data to the terminal again using a secure communication protocol. Specifically, the encoded data is encrypted and sent using the HTTPS protocol. The input to this step is the encoded sign language video data, and the output is the data sent to the terminal.

[1138] Step 9:

[1139] The terminal receives the sign language video data sent from the server and displays it on the smart glasses display. Specifically, it decodes the received data and visually displays it to the user. The input of this step is the sign language video data sent from the server, and the output is the sign language video displayed on the smart glasses display.

[1140] Step 10:

[1141] The user visually understands the sign language video displayed through the smart glasses. Specifically, the user looks at the smart glasses display and understands the instructions from the voice assistant through the sign language video. The input for this step is the sign language video displayed on the smart glasses display, and the output is the user's understanding.

[1142] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1143] This system aims to improve information access for the hearing impaired by providing high-quality real-time sign language interpretation even when a sign language interpreter is not available. In particular, by combining it with an emotion engine that recognizes the user's emotions, it realizes sign language interpretation that responds to the user's emotions.

[1144] Program processing

[1145] Audio collection and transmission

[1146] 1. The user starts speaking

[1147] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[1148] 2. The device collects the audio data

[1149] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[1150] 3. The device preprocesses the audio data

[1151] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[1152] 4. The device sends the audio data to the server

[1153] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[1154] Analysis of voice data and sign language conversion

[1155] 5. The server receives the audio data

[1156] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[1157] 6. The server converts the audio data into text data

[1158] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[1159] 7. The server processes the text data using natural language processing

[1160] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[1161] 8. The server generates sign language video using the generative model

[1162] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[1163] 9. The server recognizes the user's emotions using an emotion engine

[1164] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[1165] 10. The server adjusts the sign language video data based on emotions.

[1166] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, if the user expresses anger, the server may adjust the expression by emphasizing the sign language movements.

[1167] 11. The server encodes the sign language video data

[1168] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[1169] 12. The server sends the sign language video data to the device.

[1170] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[1171] 13. The device receives sign language video data

[1172] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[1173] 14. The device displays sign language images

[1174] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[1175] Specific examples

[1176] 1. Use in meetings

[1177] User: The meeting facilitator makes a presentation using a microphone.

[1178] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[1179] Server: Receives voice data and converts it into text using speech recognition technology. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the sign language video. After encoding, it is sent to the device.

[1180] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video to understand the content of the presentation and also grasp the presenter's emotions.

[1181] 2. Daily communication

[1182] User: Starts speaking to place an order at a cafe.

[1183] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[1184] Server: Converts voice data into text using speech recognition technology and analyzes it using natural language processing algorithms. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the video. After encoding, it is sent to the device.

[1185] Device: The received sign language video is displayed on the smart glasses. Through the video, the hearing-impaired person can understand the order details and also grasp the emotions of the store clerk.

[1186] As described above, the system of the present invention provides high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired and understanding the user's emotions.

[1187] The processing flow will be explained below.

[1188] Step 1:

[1189] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[1190] Step 2:

[1191] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[1192] Step 3:

[1193] Terminal: The collected audio data is preprocessed using noise reduction and volume normalization to improve the quality of the data. Specifically, the audio signal is processed using a digital filter to reduce background noise.

[1194] Step 4:

[1195] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with Transport Layer Security (TLS).

[1196] Step 5:

[1197] Server: Receives the audio data sent from the device, establishes a data stream, and temporarily stores the received data in local storage.

[1198] Step 6:

[1199] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[1200] Step 7:

[1201] Server: The generated text data is analyzed using a natural language processing (NLP) algorithm and converted into a format suitable for sign language. The NLP algorithm performs grammatical and semantic analysis to generate a text format that is optimal for sign language conversion.

[1202] Step 8:

[1203] Server: Converts the analyzed text data into animated sign language videos using a trained generative model. The generative model (e.g., a Transformer model) includes a neural network that inputs text data and generates sign language videos.

[1204] Step 9:

[1205] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[1206] Step 10:

[1207] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, it emphasizes the sign language movements to indicate anger, and softens the sign language movements to indicate joy.

[1208] Step 11:

[1209] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[1210] Step 12:

[1211] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with TLS.

[1212] Step 13:

[1213] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[1214] Step 14:

[1215] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[1216] As a result, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired. Furthermore, by understanding the user's emotions and providing appropriate sign language expressions, the system achieves more natural and empathetic communication.

[1217] Example 2

[1218] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1219] It is difficult for hearing-impaired people to obtain accurate and emotionally rich information in real time even in situations where a sign language interpreter is not available. Furthermore, there are an increasing number of situations where communication that includes emotional nuances, rather than just conveying information, is required. The purpose of this invention is to solve these problems and improve information access for hearing-impaired people.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1221] In this invention, the server includes means for acquiring audio data, means for preprocessing the audio data, means for converting the audio data into text data, means for generative modeling for converting the text data into sign language video data, means for adjusting the sign language video data based on a user's emotions, means for encoding the sign language video data into an appropriate multimedia format, and means for displaying the sign language video data, thereby enabling real-time, high-quality sign language interpretation and the generation of sign language video that reflects the user's emotions.

[1222] The "means for acquiring voice data" refers to a device or system that collects voices uttered by a user as digital data.

[1223] "Means for preprocessing audio data" refers to a device or system that performs processes such as noise removal and volume normalization on collected audio data to improve the quality of the data.

[1224] "Means for converting voice data into text data" refers to a device or system that uses voice recognition technology to convert voice data into text information.

[1225] "Generative model means for converting text data into sign language video data" refers to a device or system that uses a generative model algorithm to convert text information into sign language animation video.

[1226] The "means for adjusting sign language video data based on the user's emotions" refers to a device or system that analyzes the user's emotional state and adjusts the movement of the sign language video accordingly.

[1227] A "means for encoding sign language video data into a suitable multimedia format" is a device or system that converts sign language video data into a suitable multimedia format (e.g., MP4) for more efficient transmission and display.

[1228] "Means for displaying sign language video data" refers to a device or system that displays the generated and adjusted sign language video data on a device such as a monitor or smart glasses.

[1229] A "means for transmitting audio data using a secure communication protocol" is a device or system that transmits captured and pre-processed audio data to a server in a secure and reliable manner (e.g., HTTPS).

[1230] "Means for recognizing user emotions from sign language video data using an emotion engine" refers to a set of algorithms and devices for analyzing and recognizing user emotions from audio data and text data.

[1231] MODE FOR CARRYING OUT THE INVENTION

[1232] This invention provides a system that provides high-quality real-time sign language interpretation even when a sign language interpreter is not available, improving information access for the hearing impaired. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to realize sign language interpretation that responds to emotions.

[1233] Hardware and Software

[1234] Hardware:

[1235] 1. Device:

[1236] High-sensitivity microphones: used to collect audio precisely. An example is the Shure SM58.

[1237] Display: Use a monitor or smart glasses (e.g., Google Glass) to visually display the sign language image.

[1238] 2. Server:

[1239] Use servers with high-performance computing resources, such as cloud-based servers (e.g., AWS, Google Cloud).

[1240] software:

[1241] 1. Device:

[1242] Audio pre-processing software: Use the noise reduction feature in Audacity.

[1243] Communication Protocol: HTTPS and TLS are used to ensure secure communication.

[1244] 2. Server:

[1245] Speech recognition technology: Google Cloud Speech-to-Text API.

[1246] Natural language processing algorithms: SpaCy, NLTK.

[1247] Generative model: Generates sign language animation video using a Transformers model.

[1248] Sentiment Engine: IBM Watson sentiment analysis API.

[1249] Encoding tool: Use FFmpeg to encode the sign language video data into MP4 format.

[1250] Overview of program processing

[1251] 1. User:

[1252] Speak into the microphone. The system can be used in a variety of situations, such as meetings and everyday conversations.

[1253] 2. Terminal:

[1254] Collect user voice data in real time through a microphone.

[1255] Preprocess the audio data using Audacity's noise reduction function.

[1256] The preprocessed audio data is encrypted with TLS using HTTPS and sent to the server.

[1257] 3. Server:

[1258] A data stream is established to receive and temporarily store audio data sent from the terminal.

[1259] Convert the audio data into text data using the Google Cloud Speech-to-Text API.

[1260] Using SpaCy and NLTK, text data is processed using natural language processing and converted into a format suitable for sign language.

[1261] Generate animated sign language videos using Transformers models.

[1262] Using IBM Watson's sentiment analysis API, it recognizes user emotions from voice and text data.

[1263] Adjust sign language video data based on recognized emotions, for example, enhancing sign movements to indicate anger.

[1264] Use an encoding tool (FFmpeg) to encode the adjusted sign language video data into MP4 format.

[1265] The encoded sign language video data is sent to the terminal using HTTPS.

[1266] 4. Terminal:

[1267] The sign language video data transmitted from the server is received and stored in the display buffer.

[1268] The received sign language video is displayed on a monitor or smart glasses display.

[1269] Specific examples

[1270] Meeting use:

[1271] A user acts as the meeting facilitator and makes a presentation using a microphone. A device (e.g., a laptop PC) installed in the conference room collects audio in real time, preprocesses it, and sends it to a server. The server converts the received audio data into text using the Google Cloud Speech-to-Text API, and generates a sign language video using a generative model and emotion engine. The video data is encoded, sent back to the device, and displayed on a screen in the conference room. Hearing-impaired participants can understand the content of the speech and the speaker's emotions through this sign language video.

[1272] Daily communication:

[1273] A user begins speaking to place an order at a cafe. The smartphone collects the audio, preprocesses it using Audacity, and sends the audio data to a server. The server converts the audio data into text using the Google Cloud Speech-to-Text API and creates a sign language video using a generative model and emotion engine. The adjusted and encoded data is sent back to the device and displayed on the smart glasses display. The hearing-impaired user can understand the order and the waiter's emotions through this video.

[1274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1275] Step 1:

[1276] The user speaks

[1277] A user speaks into a microphone for meetings or everyday conversations. The input is the user's spoken voice. The output is an analog audio signal.

[1278] Step 2:

[1279] The device collects the audio

[1280] The device uses a high-sensitivity microphone (e.g., Shure SM58) to collect the user's voice as a digital signal. The input is an analog audio signal. The output is digital audio data.

[1281] Step 3:

[1282] The device preprocesses the audio

[1283] The device uses Audacity's noise reduction function to remove noise from the collected audio data and normalizes the volume. The input is digital audio data. The output is pre-processed, clear audio data.

[1284] Step 4:

[1285] The device sends the audio data to the server.

[1286] The device sends the preprocessed audio data to the server using HTTPS, and the TLS protocol is used for data encryption. The input is the preprocessed audio data. The output is the audio data sent to the server via a secure communication protocol.

[1287] Step 5:

[1288] The server receives the audio data.

[1289] The server receives the voice data sent from the terminal and stores it in a temporary data stream, which is used for subsequent processing. The input is the voice data sent from the terminal. The output is the temporarily stored voice data.

[1290] Step 6:

[1291] The server converts the audio data into text

[1292] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data. The output is a highly accurate text data.

[1293] Step 7:

[1294] The server processes the text in natural language.

[1295] The server analyzes the generated text data using natural language processing algorithms such as SpaCy and NLTK and converts it into a format suitable for sign language. The input is text data. The output is data in a format suitable for sign language.

[1296] Step 8:

[1297] The server generates the sign language video

[1298] The server uses a Transformers model to convert natural language processed text data into sign language animation. The input is data in a format suitable for sign language. The output is sign language animation data.

[1299] Step 9:

[1300] The server recognizes emotions

[1301] The server uses IBM Watson's sentiment analysis API to recognize the user's emotions from voice and text data. The input is voice and text data. The output is the recognized emotion information.

[1302] Step 10:

[1303] The server adjusts the sign language video based on emotions.

[1304] The server adjusts the sign language video data based on the recognized emotion. For example, if the emotion is "sadness," the sign language movements are slowed down. The input is the sign language video data and the emotion information. The output is the adjusted sign language video data.

[1305] Step 11:

[1306] The server encodes the sign language video

[1307] The server encodes the adjusted sign language video data into MP4 format using FFmpeg. The input is the adjusted sign language video data. The output is the encoded sign language video data in MP4 format.

[1308] Step 12:

[1309] The server sends the sign language video data

[1310] The server sends the encoded sign language video data to the terminal using HTTPS to ensure reliability and security of the communication. The input is the encoded sign language video data. The output is the sign language video data sent to the terminal.

[1311] Step 13:

[1312] The device receives the sign language video data.

[1313] The terminal receives sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent from the server. The output is the sign language video data stored in the display buffer.

[1314] Step 14:

[1315] The device displays a sign language video.

[1316] The device displays the received sign language video on a monitor or smart glasses (e.g., Google Glass). The input is the sign language video data stored in the display buffer. The output is the sign language video displayed on the monitor or smart glasses.

[1317] (Application example 2)

[1318] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1319] Current sign language interpretation systems have difficulty reflecting the user's emotions appropriately, making it difficult to convey emotional nuances. Furthermore, it is difficult to provide high-quality real-time sign language interpretation when a sign language interpreter is unavailable. This makes it difficult for the hearing impaired to accurately understand information.

[1320] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1321] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for generating a generative model for converting the text data into sign language video data, means for analyzing a user's emotions, means for adjusting the sign language video data based on the user's emotions, and means for displaying the sign language video data. This enables high-quality real-time sign language interpretation that reflects the user's emotions even in places where a sign language interpreter is not available, enabling people with hearing impairments to accurately understand information.

[1322] The "means for acquiring voice data" refers to a means for collecting the voice uttered by the user in digital form.

[1323] The "means for converting voice data into text data" refers to a means for converting collected voice data into text information using natural language processing technology.

[1324] The "generative model means for converting text data into sign language video data" is a means for simulating sign language movements based on character information and generating the results as video data.

[1325] The "means for displaying sign language video data" refers to a means for displaying the generated sign language video on a display device such as a monitor or smart glasses.

[1326] "Means for analyzing user emotions" refers to means for reading and analyzing the user's emotional state from voice data or text data.

[1327] The "means for adjusting sign language video data based on the user's emotions" refers to a means for adjusting the movements and expressions of the sign language video to reflect the analyzed emotions of the user.

[1328] The present invention is a system that acquires voice data, analyzes the voice data, and converts it into sign language video data in real time. Furthermore, it can realize natural and emotional communication by identifying the user's emotions and adjusting the sign language video according to those emotions. The following hardware and software are used to implement the present invention.

[1329] Hardware used

[1330] 1. Audio collection device: Use smart glasses or a smartphone equipped with a highly sensitive microphone.

[1331] 2. Display device: Use smart glasses or a monitor equipped with a display for displaying sign language images.

[1332] 3. Data processing server: A high-performance server is used to process and manage audio data and sign language video data.

[1333] Software used

[1334] 1. Speech recognition software: Uses a speech recognition API to convert voice data into text data.

[1335] 2. Natural Language Processing Algorithms: Use natural language processing (NLP) algorithms to analyze text data.

[1336] 3. Emotion Recognition Engine: An emotion recognition engine is used to analyze user emotions from voice and text.

[1337] 4. Sign language generative model: Use a generative AI model (e.g., Transformer model) that converts text into sign language.

[1338] 5. Video Encoder: Use multimedia software to encode the generated sign language video data into an appropriate format.

[1339] System Functions and Flow

[1340] 1. Audio collection: The microphone installed on the device is used to acquire the user's audio data.

[1341] 2. Speech recognition: The collected voice data is sent to the server and converted into text data using a speech recognition API.

[1342] 3. Sentiment analysis: Recognize user emotions from text data and original audio data.

[1343] 4. Sign Language Generation: Use a generative AI model to convert text data into sign language video data.

[1344] 5. Emotional adjustment: Adjust the expression of sign language video based on the recognized emotional information.

[1345] 6. Video display: After buffering, the adjusted sign language video data is displayed on the display device.

[1346] Specific examples

[1347] In a physical store, when a cashier asks a customer, "Do you have a point card?", this system allows the cashier to display a sign language image through the smart glasses. Furthermore, if the emotion of "happiness" is analyzed from the customer's attitude and facial expression, the sign language image is adjusted to reflect that emotion, resulting in more natural and human-like communication.

[1348] Prompt Sentence Examples

[1349] "Real-time sign language interpretation when a cashier asks, 'Do you have a loyalty card?'"

[1350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1351] Step 1:

[1352] (Audio collection)

[1353] The device captures the user's voice in real time through a microphone. The input is the user's speech, and the output is digital voice data. The device uses a highly sensitive microphone to improve the accuracy of the collected voice data.

[1354] Step 2:

[1355] (Preprocessing of audio data)

[1356] The device performs preprocessing on the collected voice data, such as noise removal and volume normalization, to improve the quality of the data. The input is the raw voice data acquired in step 1, and the output is clear, preprocessed voice data. This processing improves the accuracy of voice recognition.

[1357] Step 3:

[1358] (Sending audio data)

[1359] The device sends the preprocessed audio data to the server. The input is the preprocessed audio data, and the output is a secure data transfer to the server. HTTPS is used as the communication protocol, and the data is transmitted encrypted.

[1360] Step 4:

[1361] (Speech recognition of voice data)

[1362] The server converts the received voice data into text data using a voice recognition API. The input is the voice data sent from the device, and the output is the text data converted from the voice. Through this process, the content of the conversation is visualized as text.

[1363] Step 5:

[1364] (emotional analysis)

[1365] The server analyzes the user's emotions using text data and voice tone data. The input is voice data and text data, and the output is the user's emotional state (e.g., joy, anger, sadness). The emotion recognition engine analyzes the voice tone and text content.

[1366] Step 6:

[1367] (Generation of sign language video data)

[1368] The server uses a generative AI model to convert the parsed text data into sign language video data. The input is text data, and the output is basic sign language video data. The generative AI model (e.g., a Transformer model) converts the text into sign language actions.

[1369] Step 7:

[1370] (Emotional modulation of sign language video data)

[1371] The server adjusts the expression of the sign language video data based on the recognized emotion information. The input is basic sign language video data and emotion information, and the output is adjusted sign language video data. The speed and movement of the sign language are changed to reflect the emotion.

[1372] Step 8:

[1373] (Sign language video data encoding)

[1374] The server encodes the sign language video data into an appropriate multimedia format. The input is the adapted sign language video data, and the output is an encoded sign language video file. This process allows the data to be transferred efficiently.

[1375] Step 9:

[1376] (Transmission of sign language video data)

[1377] The server sends encoded sign language video data to the terminal. The input is the encoded sign language video data, and the output is the data to be sent to the terminal. HTTPS is used as the communication protocol, and the data is encrypted.

[1378] Step 10:

[1379] (Receiving and displaying sign language video data)

[1380] The terminal receives the sign language video data sent from the server and displays it on the display device. The input is the sign language video data sent from the server, and the output is a sign language video that the user can visually confirm. Buffering processing is performed to ensure that the video is displayed smoothly.

[1381] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1382] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1383] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1384] [Fourth embodiment]

[1385] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1386] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1387] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1388] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1389] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1390] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1391] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1392] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1393] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1394] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1395] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1396] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1397] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1398] This invention is a system that converts audio data into sign language video data in real time and displays it to solve the shortage of sign language interpreters and the difficulty of allocating them. Specifically, sign language interpretation is realized by acquiring audio data, converting it into text data, converting it into sign language video data using a generative model, and finally displaying the sign language video on a display means.

[1399] Program processing

[1400] Audio collection and transmission

[1401] 1. The user starts speaking

[1402] User: Speaks into the microphone. Used in meetings, lectures, and everyday conversations.

[1403] 2. The device collects the audio data

[1404] Terminal: Collects user voice data in digital format in real time via a microphone.

[1405] 3. The device preprocesses the audio data

[1406] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[1407] 4. The device sends the audio data to the server

[1408] Terminal: Sends the pre-processed audio data to the server using a secure communication protocol.

[1409] Analysis of voice data and sign language conversion

[1410] 5. The server receives the audio data

[1411] Server: Receives the audio data sent from the terminal and establishes a data stream.

[1412] 6. The server converts the audio data into text data

[1413] Server: Using voice recognition technology, the received voice data is converted into text data.

[1414] 7. The server processes the text data using natural language processing

[1415] Server: Analyzes the generated text data using a natural language processing algorithm and converts it into a format suitable for sign language.

[1416] 8. The server generates sign language video using the generative model

[1417] Server: Using a trained generative model, it generates animated sign language videos based on text data.

[1418] Sign language video generation and display

[1419] 9. The server encodes the sign language video data

[1420] Server: Encodes the generated sign language video data into an appropriate multimedia format.

[1421] 10. The server sends the sign language video data to the device.

[1422] Server: Sends encoded sign language video data to the terminal using a secure communication protocol.

[1423] 11. The device receives the sign language video data

[1424] Terminal: Receives sign language video data sent from the server and stores it in the display buffer.

[1425] 12. The device displays sign language images

[1426] Terminal: The received sign language video is displayed on a monitor or smart glasses display, allowing the user to visually understand the sign language interpretation in real time.

[1427] Specific examples

[1428] 1. Use in meetings

[1429] User: The meeting facilitator makes a presentation using a microphone.

[1430] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[1431] Server: Receives voice data and converts it into text data using speech recognition technology. Generates sign language video data using a generative model, encodes it, and sends it to the device.

[1432] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[1433] 2. Daily communication

[1434] User: Starts speaking to place an order at a cafe.

[1435] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[1436] Server: Converts voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[1437] Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[1438] As described above, the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving access to information for the hearing impaired.

[1439] The processing flow will be explained below.

[1440] Step 1:

[1441] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[1442] Step 2:

[1443] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[1444] Step 3:

[1445] Terminal: Preprocessing such as noise reduction and volume normalization is performed on the collected audio data. Here, signal processing algorithms are used to adjust the audio data so that it is clear.

[1446] Step 4:

[1447] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[1448] Step 5:

[1449] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[1450] Step 6:

[1451] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[1452] Step 7:

[1453] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[1454] Step 8:

[1455] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[1456] Step 9:

[1457] Server: Encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces the file size and improves transmission and display efficiency.

[1458] Step 10:

[1459] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[1460] Step 11:

[1461] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[1462] Step 12:

[1463] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand the smooth sign language interpretation.

[1464] As described above, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, thereby improving information access for the hearing impaired.

[1465] Example 1

[1466] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1467] Currently, there is a shortage of sign language interpreters and difficulty in locating them, which poses a major barrier for the hearing impaired, especially in situations where real-time communication is required. In addition, the technology for converting speech information into sign language is still in its infancy, and improvements in accuracy and speed are required. To solve these issues, a system that provides high-quality, real-time sign language interpretation is needed.

[1468] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1469] In this invention, the server includes a speech recognition unit that converts speech data into text data, a unit that performs natural language processing on the text data and converts it into a format suitable for sign language video data, and a unit that converts the text data into sign language video data using a generative model, thereby enabling high-quality real-time sign language interpretation.

[1470] The "means for the user to produce voice" is a system component that includes the user inputting voice into a microphone during a meeting or everyday conversation.

[1471] The "means for the terminal to collect the voice as digital voice data" refers to the function of converting analog voice into a digital signal and collecting it in real time through a microphone connected to the terminal.

[1472] The "means for preprocessing the audio data and performing noise removal and volume normalization" refers to software or hardware functions for removing unnecessary noise from the audio data and performing processing to maintain a constant volume.

[1473] The "means for transmitting preprocessed audio data to a server" is a device or method for transferring the preprocessed audio data to a server via a secure communications protocol.

[1474] The "server means for receiving the voice data" is a function of the server for receiving and processing the voice data transmitted from the terminal.

[1475] The "voice recognition means for converting the voice data into text data" refers to a technology or software function for analyzing a voice signal and converting the content into text information.

[1476] "Means for natural language processing the text data and converting it into a format suitable for sign language video data" refers to algorithms or software functions for performing natural language processing on text data, extracting and organizing the information necessary for sign language expression.

[1477] "Means for converting the text data into sign language video data using a generative model" refers to technology or software functions that convert text data into sign language animation video using a generative model.

[1478] "Means for encoding the sign language video data into a suitable multimedia format" refers to software or hardware functionality for compressing and encoding the generated sign language video data into a suitable multimedia format.

[1479] The "means for transmitting the encoded sign language video data to the receiving terminal" refers to a communication means or protocol for securely transferring the encoded sign language video data to the terminal.

[1480] The "means for displaying the received sign language video data" refers to a display device or software function for visually displaying the received sign language video data to the user on the terminal.

[1481] The "means for displaying the sign language video data on a display device" refers to hardware and software functions for displaying the sign language video data on a terminal display or an external monitor.

[1482] "Means for using a microphone and audio processing software to collect and pre-process said audio data" refers to the function of using a microphone to collect audio and pre-processing the collected audio with software.

[1483] This invention is a system that converts audio data into sign language video data in real time and displays it in order to solve the shortage of sign language interpreters and the difficulty of allocating them. How this system is implemented will be explained below in detail.

[1484] First, situations where a user may speak include a meeting or everyday conversation. The user inputs the voice through a microphone. For example, a general microphone is used in a meeting.

[1485] Next, the device, such as a laptop, collects this voice in real time as digital audio data. Specifically, a microphone connected to the device converts analog audio into a digital signal, and the collected audio data is stored in the device.

[1486] The collected voice data is pre-processed within the device, which includes noise reduction and volume normalization. The voice processing software used is, for example, general voice processing software.

[1487] The pre-processed audio data is then sent to the server using a secure communication protocol, TLS / SSL, which encrypts the data during transmission.

[1488] The server receives the voice data sent from the device. For example, a general-purpose server used as part of an online service is used as the server. The server uses the received voice data for analysis as is.

[1489] The server then converts the voice data into text data. This voice recognition process uses, for example, voice recognition technology. Specifically, a voice recognition engine analyzes the voice signal and converts it into text data.

[1490] The generated text data is then subjected to natural language processing and converted into a format suitable for sign language video data. A general natural language processing library is often used as the natural language processing algorithm. This analyzes the structure of the sentence and extracts the information necessary for sign language expression.

[1491] The converted text data is then converted into sign language video data using a generative model, such as a trained generative AI model. At this stage, the text data is converted into sign language animation video.

[1492] The resulting sign language video data is then encoded into a suitable multimedia format, for example using multimedia software, which compresses the video data and optimizes it for easy transmission and playback.

[1493] The encoded sign language video data is securely transmitted from the server to a receiving terminal, such as a smartphone carried by a user.

[1494] The receiving terminal receives the sign language video data and displays it on a display device, allowing the user to visually understand the sign language interpretation in real time.

[1495] Specific examples

[1496] Use in meetings

[1497] 1. User: The meeting facilitator makes a presentation using a standard microphone.

[1498] 2. Terminal: A laptop computer installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[1499] 3. Server: Receives the voice data and converts it into text data using speech recognition technology. It then generates sign language video data using a generative model, encodes it, and sends it to the device.

[1500] 4. Terminal: The received sign language video is displayed on the screen in the conference room in real time. Hearing-impaired participants can watch the sign language video and understand the content of the presentation.

[1501] Daily communication

[1502] 1. User: Starts speaking to place an order at a cafe.

[1503] 2. Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[1504] 3. Server: Converts the voice data into text using speech recognition technology, analyzes it using natural language processing algorithms, generates sign language video data using a generative model, encodes it, and sends it to the device.

[1505] 4. Terminal: The received sign language video is displayed on the smart glasses, allowing the hearing impaired person to understand the order details through the video.

[1506] Prompt Sentence Examples

[1507] "I would like to convert what is being said in a meeting into sign language in real time. Please tell me how to use this system."

[1508] "Please explain the setup procedure for displaying orders in sign language at a cafe."

[1509] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1510] Step 1:

[1511] The user speaks.

[1512] During meetings or everyday conversations, users input their voice into a microphone. This voice becomes the input to the system. The voice uttered by the user is sent to the terminal via the microphone.

[1513] Step 2:

[1514] The terminal collects the voice data as digital voice data.

[1515] A microphone connected to the device converts analog voice into a digital signal and collects it in real time. The collected digital voice data is stored in the device's memory. Analog voice data is the input, and digital voice data is obtained as the output.

[1516] Step 3:

[1517] The device preprocesses the audio data.

[1518] The device performs preprocessing on the collected audio data, such as noise reduction and volume normalization. Audio processing software is used to remove noise and maintain a constant volume. The input for preprocessing is digital audio data, and the output is the preprocessed audio data.

[1519] Step 4:

[1520] The terminal transmits the preprocessed audio data to the server.

[1521] The preprocessed audio data is sent to the server using a secure communication protocol (TLS / SSL). At this time, the data is encrypted and sent. The input is the preprocessed audio data, and the output is the data sent to the server.

[1522] Step 5:

[1523] The server receives the audio data.

[1524] The server receives the voice data sent from the terminal. The received voice data is stored in the server's memory. The input is the voice data sent from the terminal, and the output is the received voice data.

[1525] Step 6:

[1526] The server converts the voice data into text data.

[1527] The server uses speech recognition technology to convert the received voice data into text data. The speech recognition engine analyzes the voice signal and generates a corresponding text representation. The input is voice data and the output is text data.

[1528] Step 7:

[1529] The server performs natural language processing on the text data.

[1530] The generated text data is analyzed using natural language processing algorithms and converted into a format suitable for sign language. Sentence structure is analyzed and important information is extracted. The input is text data, and the output is data in a format suitable for sign language video data.

[1531] Step 8:

[1532] The server generates sign language video using the generative model.

[1533] Based on the generated data, a trained generative model is used to generate animated sign language videos. The input is data in a format suitable for sign language video data, and the output is sign language video data.

[1534] Step 9:

[1535] The server encodes the sign language video data.

[1536] The server encodes the generated sign language video data into an appropriate multimedia format (e.g., MP4 format). Multimedia software is used for the encoding process. The input is the sign language video data, and the output is the encoded sign language video data.

[1537] Step 10:

[1538] The server transmits the encoded sign language video data to the terminal.

[1539] The encoded sign language video data is transmitted to a terminal using a secure communication protocol, where the input is the encoded sign language video data and the output is the data transmitted to the terminal.

[1540] Step 11:

[1541] The terminal receives the sign language video data.

[1542] The terminal receives the encoded sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent to the terminal, and the output is the received sign language video data.

[1543] Step 12:

[1544] The device displays the sign language video.

[1545] The terminal displays the received sign language video data on a display device, such as a screen in a conference room or smart glasses. The input is the received sign language video data, and the output is the displayed sign language video.

[1546] (Application example 1)

[1547] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1548] Modern autonomous driving technology faces the problem of hearing-impaired people having difficulty understanding instructions from voice assistants. Furthermore, in situations where real-time sign language interpretation is not available, hearing-impaired people have difficulty obtaining information quickly and accurately. This can lead to reduced safety and convenience while driving. Therefore, there is a need for the development of a system that converts instructions from voice assistants into sign language images in real time and displays them through smart glasses.

[1549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1550] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a generative model means for converting the text data into sign language video data, thereby enabling instructions from the voice assistant to be converted into sign language video in real time and displayed through the smart glasses.

[1551] "Audio data" is a digital representation of sound collected by an acoustic sensor such as a microphone.

[1552] "Text data" is data in which voice data is expressed as character information.

[1553] A "generative model" is an artificial intelligence model that generates new data based on input data.

[1554] "Sign language video data" is video data that visually expresses actions corresponding to sign language.

[1555] A "display means" is a hardware or software device for visually presenting an image to a user.

[1556] "Smart glasses" are eyeglass-like devices worn by the user that can display information on a screen.

[1557] "Preprocessing" refers to performing quality improvement processes such as noise removal and volume normalization on the collected data.

[1558] A "multimedia format" is a digital format for uniformly handling different types of data, such as audio, video, and text data.

[1559] "Encoding" is the process of converting data into a particular format.

[1560] A "server" is a computer system for processing data.

[1561] The present invention is a system for converting voice data into sign language video data in real time and displaying it. This system is intended for use in autonomous vehicles, and displays voice instructions and navigation information from a driving assistance system as sign language video.

[1562] The system of the present invention includes the following hardware and software.

[1563] Hardware and software used

[1564] Hardware: smart glasses, microphones, in-car computer systems

[1565] Software: speech recognition engines, natural language processing libraries, generative AI models (e.g., Transformers), sign language animation generation engines, communication protocols (e.g., HTTPS)

[1566] Program processing

[1567] Imagine a scenario in which a user wearing smart glasses receives instructions from a voice assistant inside a car.

[1568] First, the user hears voice commands from the voice assistant. This voice data is collected by a microphone installed in the vehicle. The collected voice data undergoes pre-processing such as noise reduction and volume normalization to improve the data quality. The pre-processed voice data is then sent to a server via the vehicle's computer system using a secure communication protocol.

[1569] The voice data received by the server is converted into text data by a speech recognition engine (e.g., Google Speech-to-Text API). This text data is then analyzed by a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converted into a format suitable for sign language. After that, a generative AI model (e.g., GPT-3) is used to generate sign language video data based on the analyzed text data. This generated sign language video data is then encoded into an appropriate multimedia format and sent back to the in-vehicle system.

[1570] The vehicle's onboard computer system receives the sign language video data sent from the server and displays it on the smart glasses' display, allowing the user to understand voice instructions in sign language in real time.

[1571] Specific examples

[1572] Example 1: An in-car voice assistant instructs the driver to "turn right at the next traffic light." The microphone collects the audio and sends it to the server after preprocessing. The server converts the audio instruction into text and generates sign language using natural language processing. The sign language video data is displayed on the smart glasses, allowing the user to understand the instruction through sign language.

[1573] Example 2: An in-car voice assistant instructs, "Set the vehicle speed to 50 km / h." Similarly, voice data is collected, pre-processed, transmitted, and converted into text data. The generated sign language video data is displayed on the smart glasses, allowing the user to understand the instruction.

[1574] Example prompts to input to the generative AI model

[1575] "Please convert the text "Turn right at the next traffic light" into video data of sign language."

[1576] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1577] Step 1:

[1578] The device collects voice instructions using the in-car microphone. Specifically, when an instruction is given from the voice assistant, the microphone converts the analog voice data into a digital format and captures it as a data stream. The input of this step is an analog voice signal, and the output is digital voice data.

[1579] Step 2:

[1580] The digital audio data collected by the device is preprocessed, such as by noise removal and volume normalization. Specifically, an audio processing library (e.g., FFmpeg) is used to improve the quality of the collected data. The input of this step is the digital audio data, and the output is the preprocessed audio data.

[1581] Step 3:

[1582] The terminal transmits the preprocessed voice data to the server via the in-vehicle system using a secure communication protocol (e.g., HTTPS). Specifically, the encoded data is encrypted and transmitted using the HTTPS protocol. The input of this step is the preprocessed voice data, and the output is the data transmitted to the server.

[1583] Step 4:

[1584] The server converts the voice data received from the device into text data using a voice recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice recognition algorithm analyzes the voice characteristics and converts them into a corresponding string of characters. The input of this step is the voice data received by the server, and the output is text data.

[1585] Step 5:

[1586] The server analyzes the generated text data using a natural language processing (NLP) algorithm (e.g., NLTK, SpaCy) and converts it into a format suitable for sign language. Specifically, the text data is semantically analyzed and broken down into phrases that correspond to sign language. The input to this step is text data, and the output is text data in a format suitable for sign language.

[1587] Step 6:

[1588] The server uses a generative AI model (e.g., GPT-3) to generate sign language video data based on the analyzed text data. Specifically, the generative AI model generates appropriate sign language movements and outputs the data as 3D animation. The input for this step is text data in a format suitable for sign language, and the output is sign language video data.

[1589] Step 7:

[1590] The server encodes the generated sign language video data into an appropriate multimedia format. Specifically, it encodes it into a video format (e.g., MP4) and prepares it for transmission. The input of this step is the sign language video data, and the output is the encoded sign language video data.

[1591] Step 8:

[1592] The server then sends the encoded sign language video data to the terminal again using a secure communication protocol. Specifically, the encoded data is encrypted and sent using the HTTPS protocol. The input to this step is the encoded sign language video data, and the output is the data sent to the terminal.

[1593] Step 9:

[1594] The terminal receives the sign language video data sent from the server and displays it on the smart glasses display. Specifically, it decodes the received data and visually displays it to the user. The input of this step is the sign language video data sent from the server, and the output is the sign language video displayed on the smart glasses display.

[1595] Step 10:

[1596] The user visually understands the sign language video displayed through the smart glasses. Specifically, the user looks at the smart glasses display and understands the instructions from the voice assistant through the sign language video. The input for this step is the sign language video displayed on the smart glasses display, and the output is the user's understanding.

[1597] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1598] This system aims to improve information access for the hearing impaired by providing high-quality real-time sign language interpretation even when a sign language interpreter is not available. In particular, by combining it with an emotion engine that recognizes the user's emotions, it realizes sign language interpretation that responds to the user's emotions.

[1599] Program processing

[1600] Audio collection and transmission

[1601] 1. The user starts speaking

[1602] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[1603] 2. The device collects the audio data

[1604] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[1605] 3. The device preprocesses the audio data

[1606] Terminal: Performs preprocessing such as noise removal and volume normalization on the collected audio data to improve the quality of the data.

[1607] 4. The device sends the audio data to the server

[1608] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure authenticity and security of the communication.

[1609] Analysis of voice data and sign language conversion

[1610] 5. The server receives the audio data

[1611] Server: Receives the voice data sent from the terminal and establishes a data stream. Here, the received data is temporarily stored.

[1612] 6. The server converts the audio data into text data

[1613] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[1614] 7. The server processes the text data using natural language processing

[1615] Server: The generated text data is analyzed using natural language processing (NLP) algorithms and converted into a format suitable for sign language. Here, grammatical and semantic analysis is performed to generate instructions suitable for sign language movements.

[1616] 8. The server generates sign language video using the generative model

[1617] Server: Uses a trained generative model to convert the parsed text data into animated sign language videos. The generative model (e.g., a Transformer model) accurately converts the text data into animated sign language videos.

[1618] 9. The server recognizes the user's emotions using an emotion engine

[1619] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[1620] 10. The server adjusts the sign language video data based on emotions.

[1621] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, if the user expresses anger, the server may adjust the expression by emphasizing the sign language movements.

[1622] 11. The server encodes the sign language video data

[1623] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[1624] 12. The server sends the sign language video data to the device.

[1625] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). The data is encrypted to ensure reliability and security of the communication.

[1626] 13. The device receives sign language video data

[1627] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[1628] 14. The device displays sign language images

[1629] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[1630] Specific examples

[1631] 1. Use in meetings

[1632] User: The meeting facilitator makes a presentation using a microphone.

[1633] Terminal: A terminal installed in the conference room collects audio in real time, preprocesses it, and sends it to the server.

[1634] Server: Receives voice data and converts it into text using speech recognition technology. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the sign language video. After encoding, it is sent to the device.

[1635] Terminal: The received sign language video is displayed in real time on the screen in the conference room. Hearing-impaired participants can watch the sign language video to understand the content of the presentation and also grasp the presenter's emotions.

[1636] 2. Daily communication

[1637] User: Starts speaking to place an order at a cafe.

[1638] Terminal: The smartphone collects the audio, performs preprocessing, and sends the audio data to the server.

[1639] Server: Converts voice data into text using speech recognition technology and analyzes it using natural language processing algorithms. Generates sign language video data using a generative model, recognizes the user's emotions using an emotion engine, and adjusts the video. After encoding, it is sent to the device.

[1640] Device: The received sign language video is displayed on the smart glasses. Through the video, the hearing-impaired person can understand the order details and also grasp the emotions of the store clerk.

[1641] As described above, the system of the present invention provides high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired and understanding the user's emotions.

[1642] The processing flow will be explained below.

[1643] Step 1:

[1644] User: Speaks into the microphone. Used in a variety of situations, including meetings, lectures, and everyday conversations.

[1645] Step 2:

[1646] Device: Collects user voice data in real time in digital format through a microphone. This is done using a highly sensitive microphone.

[1647] Step 3:

[1648] Terminal: The collected audio data is preprocessed using noise reduction and volume normalization to improve the quality of the data. Specifically, the audio signal is processed using a digital filter to reduce background noise.

[1649] Step 4:

[1650] Terminal: The pre-processed audio data is sent to the server using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with Transport Layer Security (TLS).

[1651] Step 5:

[1652] Server: Receives the audio data sent from the device, establishes a data stream, and temporarily stores the received data in local storage.

[1653] Step 6:

[1654] Server: Using speech recognition technology, the received voice data is converted into text data. A speech recognition engine (e.g., speech recognition API) is used to generate highly accurate text.

[1655] Step 7:

[1656] Server: The generated text data is analyzed using a natural language processing (NLP) algorithm and converted into a format suitable for sign language. The NLP algorithm performs grammatical and semantic analysis to generate a text format that is optimal for sign language conversion.

[1657] Step 8:

[1658] Server: Converts the analyzed text data into animated sign language videos using a trained generative model. The generative model (e.g., a Transformer model) includes a neural network that inputs text data and generates sign language videos.

[1659] Step 9:

[1660] Server: Uses an emotion engine that analyzes the user's emotional state from voice and text data. The emotion engine determines the user's emotion (e.g., joy, anger, sadness) from the tone of the voice and the content of the text.

[1661] Step 10:

[1662] Server: Adjusts the expression of the sign language video data based on the recognized user emotion. For example, it emphasizes the sign language movements to indicate anger, and softens the sign language movements to indicate joy.

[1663] Step 11:

[1664] Server: Encode the adjusted sign language video data into an appropriate multimedia format (e.g., MP4, WebM). The encoding process reduces file size and increases transmission and display efficiency.

[1665] Step 12:

[1666] Server: The server sends the encoded sign language video data to the device using a secure communication protocol (e.g., HTTPS). To ensure reliability and security of the communication, the data is encrypted with TLS.

[1667] Step 13:

[1668] Terminal: Receives sign language video data sent from the server and stores it in the display buffer. Buffering ensures smooth video display.

[1669] Step 14:

[1670] Terminal: The received sign language video is displayed on a monitor or smart glasses display. The video is displayed in real time, allowing the user to visually understand smooth sign language interpretation that responds to emotions.

[1671] As a result, the system of the present invention can provide high-quality real-time sign language interpretation even in situations where a sign language interpreter is not available, improving information access for the hearing impaired. Furthermore, by understanding the user's emotions and providing appropriate sign language expressions, the system achieves more natural and empathetic communication.

[1672] Example 2

[1673] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1674] It is difficult for hearing-impaired people to obtain accurate and emotionally rich information in real time even in situations where a sign language interpreter is not available. Furthermore, there are an increasing number of situations where communication that includes emotional nuances, rather than just conveying information, is required. The purpose of this invention is to solve these problems and improve information access for hearing-impaired people.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1676] In this invention, the server includes means for acquiring audio data, means for preprocessing the audio data, means for converting the audio data into text data, means for generative modeling for converting the text data into sign language video data, means for adjusting the sign language video data based on a user's emotions, means for encoding the sign language video data into an appropriate multimedia format, and means for displaying the sign language video data, thereby enabling real-time, high-quality sign language interpretation and the generation of sign language video that reflects the user's emotions.

[1677] The "means for acquiring voice data" refers to a device or system that collects voices uttered by a user as digital data.

[1678] "Means for preprocessing audio data" refers to a device or system that performs processes such as noise removal and volume normalization on collected audio data to improve the quality of the data.

[1679] "Means for converting voice data into text data" refers to a device or system that uses voice recognition technology to convert voice data into text information.

[1680] "Generative model means for converting text data into sign language video data" refers to a device or system that uses a generative model algorithm to convert text information into sign language animation video.

[1681] The "means for adjusting sign language video data based on the user's emotions" refers to a device or system that analyzes the user's emotional state and adjusts the movement of the sign language video accordingly.

[1682] A "means for encoding sign language video data into a suitable multimedia format" is a device or system that converts sign language video data into a suitable multimedia format (e.g., MP4) for more efficient transmission and display.

[1683] "Means for displaying sign language video data" refers to a device or system that displays the generated and adjusted sign language video data on a device such as a monitor or smart glasses.

[1684] A "means for transmitting audio data using a secure communication protocol" is a device or system that transmits captured and pre-processed audio data to a server in a secure and reliable manner (e.g., HTTPS).

[1685] "Means for recognizing user emotions from sign language video data using an emotion engine" refers to a set of algorithms and devices for analyzing and recognizing user emotions from audio data and text data.

[1686] MODE FOR CARRYING OUT THE INVENTION

[1687] This invention provides a system that provides high-quality real-time sign language interpretation even when a sign language interpreter is not available, improving information access for the hearing impaired. In particular, by combining it with an emotion engine that recognizes the user's emotions, it is possible to realize sign language interpretation that responds to emotions.

[1688] Hardware and Software

[1689] Hardware:

[1690] 1. Device:

[1691] High-sensitivity microphones: used to collect audio precisely. An example is the Shure SM58.

[1692] Display: Use a monitor or smart glasses (e.g., Google Glass) to visually display the sign language image.

[1693] 2. Server:

[1694] Use servers with high-performance computing resources, such as cloud-based servers (e.g., AWS, Google Cloud).

[1695] software:

[1696] 1. Device:

[1697] Audio pre-processing software: Use the noise reduction feature in Audacity.

[1698] Communication Protocol: HTTPS and TLS are used to ensure secure communication.

[1699] 2. Server:

[1700] Speech recognition technology: Google Cloud Speech-to-Text API.

[1701] Natural language processing algorithms: SpaCy, NLTK.

[1702] Generative model: Generates sign language animation video using a Transformers model.

[1703] Sentiment Engine: IBM Watson sentiment analysis API.

[1704] Encoding tool: Use FFmpeg to encode the sign language video data into MP4 format.

[1705] Overview of program processing

[1706] 1. User:

[1707] Speak into the microphone. The system can be used in a variety of situations, such as meetings and everyday conversations.

[1708] 2. Terminal:

[1709] Collect user voice data in real time through a microphone.

[1710] Preprocess the audio data using Audacity's noise reduction function.

[1711] The preprocessed audio data is encrypted with TLS using HTTPS and sent to the server.

[1712] 3. Server:

[1713] A data stream is established to receive and temporarily store audio data sent from the terminal.

[1714] Convert the audio data into text data using the Google Cloud Speech-to-Text API.

[1715] Using SpaCy and NLTK, text data is processed using natural language processing and converted into a format suitable for sign language.

[1716] Generate animated sign language videos using Transformers models.

[1717] Using IBM Watson's sentiment analysis API, it recognizes user emotions from voice and text data.

[1718] Adjust sign language video data based on recognized emotions, for example, enhancing sign movements to indicate anger.

[1719] Use an encoding tool (FFmpeg) to encode the adjusted sign language video data into MP4 format.

[1720] The encoded sign language video data is sent to the terminal using HTTPS.

[1721] 4. Terminal:

[1722] The sign language video data transmitted from the server is received and stored in the display buffer.

[1723] The received sign language video is displayed on a monitor or smart glasses display.

[1724] Specific examples

[1725] Meeting use:

[1726] A user acts as the meeting facilitator and makes a presentation using a microphone. A device (e.g., a laptop PC) installed in the conference room collects audio in real time, preprocesses it, and sends it to a server. The server converts the received audio data into text using the Google Cloud Speech-to-Text API, and generates a sign language video using a generative model and emotion engine. The video data is encoded, sent back to the device, and displayed on a screen in the conference room. Hearing-impaired participants can understand the content of the speech and the speaker's emotions through this sign language video.

[1727] Daily communication:

[1728] A user begins speaking to place an order at a cafe. The smartphone collects the audio, preprocesses it using Audacity, and sends the audio data to a server. The server converts the audio data into text using the Google Cloud Speech-to-Text API and creates a sign language video using a generative model and emotion engine. The adjusted and encoded data is sent back to the device and displayed on the smart glasses display. The hearing-impaired user can understand the order and the waiter's emotions through this video.

[1729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1730] Step 1:

[1731] The user speaks

[1732] A user speaks into a microphone for meetings or everyday conversations. The input is the user's spoken voice. The output is an analog audio signal.

[1733] Step 2:

[1734] The device collects the audio

[1735] The device uses a high-sensitivity microphone (e.g., Shure SM58) to collect the user's voice as a digital signal. The input is an analog audio signal. The output is digital audio data.

[1736] Step 3:

[1737] The device preprocesses the audio

[1738] The device uses Audacity's noise reduction function to remove noise from the collected audio data and normalizes the volume. The input is digital audio data. The output is pre-processed, clear audio data.

[1739] Step 4:

[1740] The device sends the audio data to the server.

[1741] The device sends the preprocessed audio data to the server using HTTPS, and the TLS protocol is used for data encryption. The input is the preprocessed audio data. The output is the audio data sent to the server via a secure communication protocol.

[1742] Step 5:

[1743] The server receives the audio data.

[1744] The server receives the voice data sent from the terminal and stores it in a temporary data stream, which is used for subsequent processing. The input is the voice data sent from the terminal. The output is the temporarily stored voice data.

[1745] Step 6:

[1746] The server converts the audio data into text

[1747] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the voice data. The output is a highly accurate text data.

[1748] Step 7:

[1749] The server processes the text in natural language.

[1750] The server analyzes the generated text data using natural language processing algorithms such as SpaCy and NLTK and converts it into a format suitable for sign language. The input is text data. The output is data in a format suitable for sign language.

[1751] Step 8:

[1752] The server generates the sign language video

[1753] The server uses a Transformers model to convert natural language processed text data into sign language animation. The input is data in a format suitable for sign language. The output is sign language animation data.

[1754] Step 9:

[1755] The server recognizes emotions

[1756] The server uses IBM Watson's sentiment analysis API to recognize the user's emotions from voice and text data. The input is voice and text data. The output is the recognized emotion information.

[1757] Step 10:

[1758] The server adjusts the sign language video based on emotions.

[1759] The server adjusts the sign language video data based on the recognized emotion. For example, if the emotion is "sadness," the sign language movements are slowed down. The input is the sign language video data and the emotion information. The output is the adjusted sign language video data.

[1760] Step 11:

[1761] The server encodes the sign language video

[1762] The server encodes the adjusted sign language video data into MP4 format using FFmpeg. The input is the adjusted sign language video data. The output is the encoded sign language video data in MP4 format.

[1763] Step 12:

[1764] The server sends the sign language video data

[1765] The server sends the encoded sign language video data to the terminal using HTTPS to ensure reliability and security of the communication. The input is the encoded sign language video data. The output is the sign language video data sent to the terminal.

[1766] Step 13:

[1767] The device receives the sign language video data.

[1768] The terminal receives sign language video data sent from the server and stores it in a display buffer. The input is the sign language video data sent from the server. The output is the sign language video data stored in the display buffer.

[1769] Step 14:

[1770] The device displays a sign language video.

[1771] The device displays the received sign language video on a monitor or smart glasses (e.g., Google Glass). The input is the sign language video data stored in the display buffer. The output is the sign language video displayed on the monitor or smart glasses.

[1772] (Application example 2)

[1773] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1774] Current sign language interpretation systems have difficulty reflecting the user's emotions appropriately, making it difficult to convey emotional nuances. Furthermore, it is difficult to provide high-quality real-time sign language interpretation when a sign language interpreter is unavailable. This makes it difficult for the hearing impaired to accurately understand information.

[1775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1776] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for generating a generative model for converting the text data into sign language video data, means for analyzing a user's emotions, means for adjusting the sign language video data based on the user's emotions, and means for displaying the sign language video data. This enables high-quality real-time sign language interpretation that reflects the user's emotions even in places where a sign language interpreter is not available, enabling people with hearing impairments to accurately understand information.

[1777] The "means for acquiring voice data" refers to a means for collecting the voice uttered by the user in digital form.

[1778] The "means for converting voice data into text data" refers to a means for converting collected voice data into text information using natural language processing technology.

[1779] The "generative model means for converting text data into sign language video data" is a means for simulating sign language movements based on character information and generating the results as video data.

[1780] The "means for displaying sign language video data" refers to a means for displaying the generated sign language video on a display device such as a monitor or smart glasses.

[1781] "Means for analyzing user emotions" refers to means for reading and analyzing the user's emotional state from voice data or text data.

[1782] The "means for adjusting sign language video data based on the user's emotions" refers to a means for adjusting the movements and expressions of the sign language video to reflect the analyzed emotions of the user.

[1783] The present invention is a system that acquires voice data, analyzes the voice data, and converts it into sign language video data in real time. Furthermore, it can realize natural and emotional communication by identifying the user's emotions and adjusting the sign language video according to those emotions. The following hardware and software are used to implement the present invention.

[1784] Hardware used

[1785] 1. Audio collection device: Use smart glasses or a smartphone equipped with a highly sensitive microphone.

[1786] 2. Display device: Use smart glasses or a monitor equipped with a display for displaying sign language images.

[1787] 3. Data processing server: A high-performance server is used to process and manage audio data and sign language video data.

[1788] Software used

[1789] 1. Speech recognition software: Uses a speech recognition API to convert voice data into text data.

[1790] 2. Natural Language Processing Algorithms: Use natural language processing (NLP) algorithms to analyze text data.

[1791] 3. Emotion Recognition Engine: An emotion recognition engine is used to analyze user emotions from voice and text.

[1792] 4. Sign language generative model: Use a generative AI model (e.g., Transformer model) that converts text into sign language.

[1793] 5. Video Encoder: Use multimedia software to encode the generated sign language video data into an appropriate format.

[1794] System Functions and Flow

[1795] 1. Audio collection: The microphone installed on the device is used to acquire the user's audio data.

[1796] 2. Speech recognition: The collected voice data is sent to the server and converted into text data using a speech recognition API.

[1797] 3. Sentiment analysis: Recognize user emotions from text data and original audio data.

[1798] 4. Sign Language Generation: Use a generative AI model to convert text data into sign language video data.

[1799] 5. Emotional adjustment: Adjust the expression of sign language video based on the recognized emotional information.

[1800] 6. Video display: After buffering, the adjusted sign language video data is displayed on the display device.

[1801] Specific examples

[1802] In a physical store, when a cashier asks a customer, "Do you have a point card?", this system allows the cashier to display a sign language image through the smart glasses. Furthermore, if the emotion of "happiness" is analyzed from the customer's attitude and facial expression, the sign language image is adjusted to reflect that emotion, resulting in more natural and human-like communication.

[1803] Prompt Sentence Examples

[1804] "Real-time sign language interpretation when a cashier asks, 'Do you have a loyalty card?'"

[1805] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1806] Step 1:

[1807] (Audio collection)

[1808] The device captures the user's voice in real time through a microphone. The input is the user's speech, and the output is digital voice data. The device uses a highly sensitive microphone to improve the accuracy of the collected voice data.

[1809] Step 2:

[1810] (Preprocessing of audio data)

[1811] The device performs preprocessing on the collected voice data, such as noise removal and volume normalization, to improve the quality of the data. The input is the raw voice data acquired in step 1, and the output is clear, preprocessed voice data. This processing improves the accuracy of voice recognition.

[1812] Step 3:

[1813] (Sending audio data)

[1814] The device sends the preprocessed audio data to the server. The input is the preprocessed audio data, and the output is a secure data transfer to the server. HTTPS is used as the communication protocol, and the data is transmitted encrypted.

[1815] Step 4:

[1816] (Speech recognition of voice data)

[1817] The server converts the received voice data into text data using a voice recognition API. The input is the voice data sent from the device, and the output is the text data converted from the voice. Through this process, the content of the conversation is visualized as text.

[1818] Step 5:

[1819] (emotional analysis)

[1820] The server analyzes the user's emotions using text data and voice tone data. The input is voice data and text data, and the output is the user's emotional state (e.g., joy, anger, sadness). The emotion recognition engine analyzes the voice tone and text content.

[1821] Step 6:

[1822] (Generation of sign language video data)

[1823] The server uses a generative AI model to convert the parsed text data into sign language video data. The input is text data, and the output is basic sign language video data. The generative AI model (e.g., a Transformer model) converts the text into sign language actions.

[1824] Step 7:

[1825] (Emotional modulation of sign language video data)

[1826] The server adjusts the expression of the sign language video data based on the recognized emotion information. The input is basic sign language video data and emotion information, and the output is adjusted sign language video data. The speed and movement of the sign language are changed to reflect the emotion.

[1827] Step 8:

[1828] (Sign language video data encoding)

[1829] The server encodes the sign language video data into an appropriate multimedia format. The input is the adapted sign language video data, and the output is an encoded sign language video file. This process allows the data to be transferred efficiently.

[1830] Step 9:

[1831] (Transmission of sign language video data)

[1832] The server sends encoded sign language video data to the terminal. The input is the encoded sign language video data, and the output is the data to be sent to the terminal. HTTPS is used as the communication protocol, and the data is encrypted.

[1833] Step 10:

[1834] (Receiving and displaying sign language video data)

[1835] The terminal receives the sign language video data sent from the server and displays it on the display device. The input is the sign language video data sent from the server, and the output is a sign language video that the user can visually confirm. Buffering processing is performed to ensure that the video is displayed smoothly.

[1836] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1837] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1838] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1839] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1840] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1841] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1842] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1843] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1844] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1845] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1846] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1847] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1848] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1849] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1850] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1851] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1852] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1853] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1854] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1855] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1856] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1857] The following is further disclosed regarding the above embodiment.

[1858] (Claim 1)

[1859] means for acquiring audio data;

[1860] means for converting the voice data into text data;

[1861] a generative model means for converting the text data into sign language video data;

[1862] means for displaying the sign language video data;

[1863] A system including:

[1864] (Claim 2)

[1865] The system of claim 1 , further comprising means for preprocessing the audio data.

[1866] (Claim 3)

[1867] 10. The system of claim 1, further comprising: means for encoding the sign language video data into a suitable multimedia format.

[1868] (Claim 4)

[1869] 10. The system of claim 1, further comprising means for transmitting the sign language video data over a secure communication protocol.

[1870] "Example 1"

[1871] (Claim 1)

[1872] a means for a user to produce a sound;

[1873] a means for collecting the voice in the form of digital voice data in the terminal;

[1874] means for pre-processing the audio data to perform noise removal and volume normalization;

[1875] means for transmitting the preprocessed audio data to a server;

[1876] a server means for receiving the voice data;

[1877] a voice recognition means for converting the voice data into text data;

[1878] means for performing natural language processing on the text data and converting it into a format suitable for sign language video data;

[1879] means for converting the text data into sign language video data using a generative model;

[1880] means for encoding said sign language video data into a suitable multimedia format;

[1881] means for transmitting the encoded sign language video data to a receiving terminal;

[1882] means for displaying the received sign language video data;

[1883] A system including:

[1884] (Claim 2)

[1885] 10. The system of claim 1, further comprising: means for displaying the sign language video data on a display device.

[1886] (Claim 3)

[1887] 10. The system of claim 1, further comprising means for using a microphone and audio processing software to collect and pre-process said audio data.

[1888] "Application Example 1"

[1889] (Claim 1)

[1890] means for acquiring audio data;

[1891] means for converting the voice data into text data;

[1892] a generative model means for converting the text data into sign language video data;

[1893] means for displaying the sign language video data;

[1894] The display means includes a smart glass;

[1895] A system including:

[1896] (Claim 2)

[1897] The system of claim 1 , further comprising means for preprocessing the audio data.

[1898] (Claim 3)

[1899] 10. The system of claim 1, further comprising: means for encoding the sign language video data into a suitable multimedia format.

[1900] "Example 2: Combining Emotion Engines"

[1901] (Claim 1)

[1902] means for acquiring audio data;

[1903] means for preprocessing the audio data;

[1904] means for converting the voice data into text data;

[1905] a generative model means for converting the text data into sign language video data;

[1906] means for adjusting the sign language video data based on a user's emotion;

[1907] means for encoding said sign language video data into a suitable multimedia format;

[1908] means for displaying the sign language video data;

[1909] A system including:

[1910] (Claim 2)

[1911] 10. The system of claim 1, further comprising means for transmitting the audio data in a secure communication protocol.

[1912] (Claim 3)

[1913] The system of claim 1 , further comprising: means for recognizing a user's emotion in the sign language video data with an emotion engine.

[1914] "Application example 2 when combining emotion engines"

[1915] (Claim 1)

[1916] means for acquiring audio data;

[1917] means for converting the voice data into text data;

[1918] a generative model means for converting the text data into sign language video data;

[1919] means for displaying the sign language video data;

[1920] means for analyzing user emotions;

[1921] means for adjusting the sign language video data based on a user's emotion;

[1922] A system including:

[1923] (Claim 2)

[1924] The system of claim 1 , further comprising means for preprocessing the audio data.

[1925] (Claim 3)

[1926] 10. The system of claim 1, further comprising: means for encoding the sign language video data into a suitable multimedia format. [Explanation of symbols]

[1927] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring audio data; means for converting the voice data into text data; a generative model means for converting the text data into sign language video data; means for displaying the sign language video data; A system including:

2. The system of claim 1 further comprising means for preprocessing the audio data.

3. The system of claim 1 further comprising means for encoding said sign language video data into a suitable multimedia format.

4. The system of claim 1 , further comprising means for transmitting the sign language video data in a secure communication protocol.

Images