system

Abstract

We provide the system. [Solution] A means of acquiring video data for recognizing sign language, A method for extracting the characteristics of sign language from acquired video data, A means of analyzing extracted features and converting them into corresponding natural language, A means of outputting the converted natural language data, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In communication between a hearing - impaired person and a healthy person, there is a problem that it is difficult to smoothly communicate with a healthy person who does not understand sign language. Also, due to the lack of technology for accurately converting sign language into voice or text while maintaining real - time performance, there is a problem that information cannot be smoothly transmitted in daily life or workplaces.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for acquiring video data for recognizing sign language, means for extracting sign language features from the acquired video data, means for analyzing the extracted features and converting them into corresponding natural language, and means for outputting the converted natural language data. Furthermore, by including means for acquiring audio data and converting it into natural language, means for converting the converted natural language into sign language motion data, and means for outputting the converted sign language motion data, two-way real-time communication is enabled. This system has learning means for learning the characteristics of each user and improving the conversion accuracy, thereby achieving highly accurate translation tailored to individual needs.

[0006] Sign language is a visual form of language used by people with hearing impairments, and it is a means of communication through the movements, positions, and facial expressions of the hands and fingers.

[0007] "Video data" refers to data that stores visual information acquired by a camera or other recording device in digital format.

[0008] "Characteristics" refer to important elements in sign language recognition, such as hand position, movement, shape, and finger placement.

[0009] "Natural language" refers to the language that humans use for normal communication, and which can be expressed as spoken or written language.

[0010] "Audio data" refers to data that digitally stores sound waves recorded from human speech, and is the target of speech recognition.

[0011] "Learning methods" refer to functions and methods that allow a system to analyze the individual characteristics of users based on past data and improve accuracy.

[0012] "Real-time" refers to a process where the delay between user input and output is extremely short, meaning processing occurs almost instantly. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a tagged processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a tagged storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The real-time sign language translation system according to the present invention is a system that enables smooth communication between hearing individuals who do not understand sign language and hearing-impaired individuals. This system acquires sign language video in real time and converts it into speech or text to convey the intended message. It also has the function of converting the speech of hearing individuals into sign language and conveying it visually.

[0035] The device captures the user's sign language in real time via its camera. The video data is transmitted to the server with low latency. The server analyzes the received data and extracts the characteristics of the sign language. This identifies which sign language was shown and converts it into corresponding natural language text. The converted text is output as speech using speech synthesis technology and played back on the device.

[0036] Conversely, the device acquires the voice spoken by a hearing person using a microphone and sends that voice data to a server. The server converts the voice data into text using speech recognition technology, and then converts the text into appropriate sign language movements. These sign language movements are displayed as animations on the device, conveying the intended message to the hearing impaired.

[0037] As a concrete example, if a user (who is hearing impaired) expresses "Good morning" in sign language, the terminal sends the video to the server, which analyzes the sign language and converts it into the text "Good morning." This text is then output as audio and instantly transmitted to a hearing person.

[0038] This system goes beyond simple sign language-to-speech conversion; it has the ability to learn each user's individual sign language style and speech patterns, improving its conversion accuracy over time. This enables highly accurate translations, making it useful in a variety of communication situations.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The device captures the user's sign language in real time via its camera. It acquires the video at a high frame rate and stores the visual information as digital data.

[0042] Step 2:

[0043] The terminal compresses and encodes the captured video data and transmits it to the server via the communication line with low latency.

[0044] Step 3:

[0045] The server extracts features such as hand position, movement, and finger shape from the received video data. It uses computer vision algorithms to identify the necessary feature points.

[0046] Step 4:

[0047] The server uses a deep learning model to recognize sign language based on the extracted feature points. Using the trained model, it generates corresponding natural language text data.

[0048] Step 5:

[0049] The server uses natural language processing technology to generate synthesized speech from the obtained text data. This creates audio output as audio data.

[0050] Step 6:

[0051] The server sends the generated audio data to the terminal.

[0052] Step 7:

[0053] The device plays the received audio data through its speaker, conveying the content of the sign language to a hearing person as audio.

[0054] Step 8:

[0055] The device acquires the voice spoken by a healthy person through its microphone and sends the voice data to the server.

[0056] Step 9:

[0057] The server converts the received audio data into text using speech recognition technology. By obtaining the text data, it prepares for the next processing step.

[0058] Step 10:

[0059] The server converts the obtained text data into sign language motion data. It then refers to a sign language database to generate specific sign language animations.

[0060] Step 11:

[0061] The server sends the generated sign language animation data to the terminal.

[0062] Step 12:

[0063] The device displays received sign language animations on its screen, visually conveying messages to people with hearing impairments.

[0064] (Example 1)

[0065] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0066] Facilitating smooth communication between deaf individuals who use sign language and hearing individuals who do not understand sign language is difficult. Conventional systems struggle to provide accurate real-time translation, thus creating a need for technology that efficiently performs bidirectional conversion between sign language and speech or text.

[0067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0068] In this invention, the server includes a device for acquiring video information, means for extracting sign language features from the acquired video information, and a mechanism for analyzing the extracted features and converting them into corresponding natural language. This makes it possible to convert sign language into natural language in real time with high accuracy and output it as speech.

[0069] "Visual information" refers to dynamic visual data acquired to represent sign language movements.

[0070] "Characteristics of sign language" refer to identifiable elements such as the shape, movement, and position of the hands in sign language actions.

[0071] "Natural language" refers to the language that humans use for normal communication, and which can be expressed as spoken or written language.

[0072] "Synthesized speech" is a technology that converts natural language text information into audio signals and plays them back using mechanically generated speech.

[0073] "Audio information" refers to data obtained from speech and is used for speech recognition processing.

[0074] "Sign language motion information" refers to the motion data that makes up sign language animations generated based on audio and text.

[0075] "Animation" refers to a series of visual representations used to visually represent sign language and its movements.

[0076] "Individual user characteristics" refers to the unique actions and vocal patterns of each user in sign language and speech.

[0077] A "learning device" is a processing device that continuously learns the characteristics of the user's sign language and speech to improve conversion accuracy.

[0078] The real-time sign language translation system according to this invention aims to enable smooth communication between hearing-impaired individuals who use sign language and hearing individuals who do not understand sign language. Specific embodiments for carrying out this invention are described below.

[0079] The device uses a camera to capture the user's sign language movements in real time. The video information is processed by specialized equipment such as a Movidius Neural Compute Stick and transmitted to a server with low latency. The video information acquired here is high-quality visual data for accurately capturing sign language movements.

[0080] The server analyzes the received video information using a deep learning model. A CNN model utilizing TENSORFLOW® is effective as the model, and this model functions as a mechanism to extract sign language features and convert them into corresponding natural language. In this process, the server identifies sign language features such as hand shape, movement, and position. The natural language converted based on the analysis is output as synthesized speech using speech synthesis technology such as Google® Text-to-Speech API.

[0081] Conversely, voice information spoken by a healthy person through the microphone on the device is also sent to the server. The server uses the Google Speech-to-Text API to convert this voice information into natural language text. This is then converted into sign language gesture information and visually displayed on the device using animation software such as Unity3D.

[0082] As a concrete example, if a user (who is hearing impaired) expresses "thank you" in sign language, the device sends the video to the server. The server analyzes the sign language, converts it into natural language text "thank you," and outputs it as synthesized speech to a hearing person.

[0083] One possible prompt to input into the generative AI model is: "Capture a video using a specific sign language action as an example, and explain how it is converted into speech."

[0084] This system also has a function that improves conversion accuracy by learning the user's individual sign language style and voice patterns over time, and is expected to be used in an even wider range of communication scenarios.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The device captures the user's sign language movements in real time via its camera. During this process, the device acquires video information and records the images at a high frame rate. The input is the user's sign language movements, and the output is raw video data.

[0088] Step 2:

[0089] The terminal compresses the acquired video information into the required format and sends it to the server. Here, the video is optimized to streamline data transfer and enable low-latency communication. The input is raw video data, and the output is compressed video data.

[0090] Step 3:

[0091] The server receives the transmitted compressed video data and extracts sign language features using a deep learning model. Specifically, a model using TensorFlow analyzes the shape and movement of the hands and recognizes feature points based on that analysis. The input is compressed video data, and the output is sign language feature information.

[0092] Step 4:

[0093] The server utilizes a sign language-to-text mapping database to convert extracted sign language feature information into natural language. This analysis converts the intent conveyed by the sign language into natural language sentences. The input is sign language feature information, and the output is natural language text data.

[0094] Step 5:

[0095] The server converts the translated natural language text into speech data using speech synthesis technology. The Google Text-to-Speech API is used for the speech generation process. The input is natural language text data, and the output is synthesized speech.

[0096] Step 6:

[0097] The device receives synthesized speech transmitted from the server and plays the speech through its built-in speaker. This allows hearing individuals to understand the meaning of sign language as spoken language. The input is synthesized speech data, and the output is spoken audio.

[0098] Step 7:

[0099] The device captures voice information spoken by a healthy individual using a microphone. The acquired voice information is recorded in clear quality. The input is the voice of a healthy individual, and the output is voice data.

[0100] Step 8:

[0101] The terminal optimizes and compresses the recorded audio data before sending it to the server. It then transmits it with low latency. The input is audio data, and the output is compressed audio data.

[0102] Step 9:

[0103] The server converts the received audio data into natural language text using speech recognition technology. The Google Speech-to-Text API analyzes the content of the audio and converts it into text format. The input is compressed audio data, and the output is text data.

[0104] Step 10:

[0105] The server converts text data into sign language motion information. In this process, it uses an animation engine to convert the text into motion data in order to generate corresponding sign language animations. The input is text data, and the output is sign language motion information.

[0106] Step 11:

[0107] The terminal displays sign language gesture information received from the server as an animation on its screen. This allows the user (a person with hearing impairment) to receive messages visually. The input is sign language gesture information, and the output is a sign language animation.

[0108] (Application Example 1)

[0109] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0110] In modern society, the lack of effective means of smooth communication between the hearing impaired and hearing individuals remains a significant challenge. For the hearing impaired, whose primary means of communication is sign language, communication with hearing individuals is often difficult in many situations in daily life. Therefore, there is a need for real-time conversion between sign language and spoken language to achieve smooth and accurate communication. Furthermore, providing conversion accuracy optimized for each individual user is also crucial.

[0111] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0112] In this invention, the server includes means for acquiring video data for recognizing sign language, means for acquiring audio data and converting it into natural language, and means for learning user characteristics using feedback information and improving conversion accuracy. This enables real-time and highly accurate communication between hearing-impaired and hearing individuals.

[0113] "Means for acquiring video data for recognizing sign language" refers to methods for recording sign language movements in real time using a camera mounted on a terminal or device.

[0114] "Methods for extracting the characteristics of sign language" refer to algorithms that identify specific hand shapes and movement patterns from acquired sign language video data and analyze the meaning of the sign language.

[0115] "Means for converting to corresponding natural language" refers to methods for converting extracted sign language features into words and treating them as text data with concrete meaning.

[0116] "Means for converting converted natural language data into speech and outputting it via a speech output device" refers to a technology for converting generated text data into sound waves using speech synthesis technology and then playing them back through an acoustic device.

[0117] "Means for acquiring audio data and converting it into natural language" refers to speech recognition technology that analyzes input audio data and expresses its content as text.

[0118] "Means for converting natural language into sign language motion data" refers to algorithms for converting natural language text information into sequences of actions that can be communicated in sign language.

[0119] "Means for displaying sign language motion data on a visual display device" refers to a display technology that animates generated sign language motions and presents them visually.

[0120] "Methods for learning user characteristics using feedback information and improving conversion accuracy" refers to a function that analyzes the user's usage history and feedback data, optimizes the conversion algorithm, and achieves highly accurate translations tailored to individual users.

[0121] This invention is a system for facilitating communication using sign language. The system mainly consists of a terminal worn by the user and a server connected to it.

[0122] The devices are smart glasses or other mobile devices. They have built-in cameras and microphones that capture the user's sign language and voice, respectively. The sign language video data captured by the camera is transmitted to a server with low latency. The server analyzes the acquired video using machine learning models such as TensorFlow Lite to extract sign language features. This analysis identifies the meaning of the sign language movements. The identified sign language features are converted into natural language text and then spoken using speech synthesis technology. This allows the information to be conveyed to hearing individuals in both text and audio formats.

[0123] Conversely, voice input from able-bodied individuals is captured by the device and sent to the server. The server uses the Google Cloud Speech-to-Text API to convert this voice data into natural language text. Then, a generative AI model is used to convert the text into sign language motion data. This sign language motion is then animated using software such as Unity and displayed on the device's screen.

[0124] Furthermore, the system learns user characteristics and improves translation accuracy by receiving feedback as needed. This enables the achievement of highly accurate translations optimized for individual users.

[0125] For example, if a user (who is hearing impaired) uses sign language to mean "turn up the TV volume," the server analyzes this sign language and converts it into the natural language phrase "Please turn up the TV volume." Conversely, if a hearing person says "Good morning," the system animates this speech as sign language and conveys it to the user. An example of a prompt message to the generative AI model would be: "Analyze the sign language actions captured by the user via the camera and output their intent as spoken language. Also, display the spoken language as a sign language animation on the screen."

[0126] This system makes it possible for both people with hearing impairments and those without hearing impairments to communicate without difficulty.

[0127] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0128] Step 1:

[0129] The device captures the user's sign language expressions with its camera and temporarily stores the video data. Next, it transmits this video data to the server with low latency. The input is the user's sign language actions, and the output is the transfer of video data to the server.

[0130] Step 2:

[0131] The server uses TensorFlow Lite to extract sign language features from the received video data. The input is captured video data of sign language, and the output is a sign language feature vector. This vector is used to analyze the content of the sign language and convert it into natural language text.

[0132] Step 3:

[0133] The server converts the generated natural language text into speech using speech synthesis technology (e.g., a text-to-speech engine). The input is natural language text, and the output is audio data. This audio data is sent to the terminal and played back via an audio output device.

[0134] Step 4:

[0135] In the reverse process, the terminal acquires voice from a healthy person using a microphone and sends that voice data to the server. The input is the voice of the healthy person, and the output is the transfer of voice data to the server.

[0136] Step 5:

[0137] The server uses the Google Cloud Speech-to-Text API to convert speech data into natural language text. The input is speech data from a healthy individual, and the output is natural language text. Based on this text, a generative AI model is used to convert the text into sign language gesture data.

[0138] Step 6:

[0139] The server animates the converted sign language motion data using software such as Unity, and sends this animated data to the terminal. The input is the sign language motion data, and the output is the transmission of the animated sign language data to the terminal.

[0140] Step 7:

[0141] The terminal displays animated sign language to the user. The input is animation data from the server, and the output is sign language information visually communicated to the user.

[0142] Step 8:

[0143] Based on user and healthy user feedback, the server updates the conversion model and learns to achieve optimized conversion accuracy for each user. The input is the feedback data and existing model parameters, and the output is the optimized conversion model.

[0144] This process enables real-time translation between sign language and spoken language, thereby lowering communication barriers.

[0145] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0146] This invention aims to achieve richer communication by adding a function to recognize user emotions to a system that performs mutual conversion between sign language and natural language. This system is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice during the process of extracting the characteristics of sign language, and can convert them into natural language text while taking those emotions into consideration.

[0147] The device captures the user's sign language and facial expressions using its camera. The video data is sent to a server, where characteristic hand and facial movements are analyzed. In addition to the content of the sign language, the server uses an emotion engine to recognize the user's emotions, identifying expressions such as smiles, anger, and sadness. This extracts the nuances of emotion conveyed in the spoken content.

[0148] Based on the analyzed data, the server generates natural language text that reflects emotions and outputs it as synthesized speech. The terminal plays the received audio through its speaker, conveying the content of the sign language and the emotions behind it as audio information to a hearing person. For example, if a user signs "hello" with a smile, the audio output will say "hello" in a cheerful tone.

[0149] Furthermore, when the device acquires the voice of a healthy person, the server converts the voice into text and generates appropriate sign language actions with corresponding emotions from that text. The generated sign language data is sent to the device as an animation and displayed to the user. At this time, based on the emotion data, for example, a word pronounced with emotion, such as "wonderful," will be reproduced with a sign language animation that conveys the appropriate emotion.

[0150] By incorporating emotion recognition in this way, it becomes possible to enable more emotionally rich two-way communication between natural language and sign language than before. This feature will be particularly useful for users in situations where complex emotions need to be conveyed, or in informal communication.

[0151] The following describes the processing flow.

[0152] Step 1:

[0153] The device uses a camera to simultaneously capture the user's sign language movements and facial expressions. High-resolution video data is recorded in real time and transmitted to a server via the network.

[0154] Step 2:

[0155] The server extracts features corresponding to sign language movements from the received video data. In addition to hand position and movement, it analyzes facial expression data to capture emotional elements along with the sign language.

[0156] Step 3:

[0157] The server uses an emotion engine to recognize the user's emotions from the extracted facial expression data. It identifies the type of emotion (e.g., joy, sadness, surprise, etc.) and analyzes this information in conjunction with the characteristics of sign language.

[0158] Step 4:

[0159] The server generates corresponding natural language text based on the characteristics and emotional information of the sign language. The emotional information is reflected in the text to ensure that the expression includes emotional nuances.

[0160] Step 5:

[0161] The server converts the generated text data into synthesized speech. Based on emotional information, it adjusts the tone and speed of the speech to create speech data that conveys emotion.

[0162] Step 6:

[0163] The server sends synthesized speech data to the terminal, and the terminal plays the received speech to the user through its speaker. A healthy person receives the emotionally charged sign language content as speech.

[0164] Step 7:

[0165] The device captures speech uttered by a healthy individual using its microphone and sends that audio data to a server.

[0166] Step 8:

[0167] The server uses speech recognition technology to convert the audio data into text, analyzes its content, and estimates the emotions conveyed.

[0168] Step 9:

[0169] The server generates sign language action data with appropriate emotional expressions based on estimated emotional information. It then creates sign language animations that reflect emotions by referring to a sign language database.

[0170] Step 10:

[0171] The server sends the generated sign language animation data to the terminal.

[0172] Step 11:

[0173] The device displays sign language animations on its screen, conveying emotions through sign language in a way that is visible to the hearing impaired. As a result, users can visually understand emotions along with the audio content.

[0174] (Example 2)

[0175] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0176] Conventional sign language translation systems failed to consider the user's emotions when converting sign language movements into natural language, resulting in a lack of emotional transmission in communication. Furthermore, even when converting spoken language from hearing individuals into sign language, generating sign language movements that reflected appropriate emotions was difficult. This made it challenging to accurately convey intentions, especially in conversations where emotional nuances are crucial.

[0177] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0178] In this invention, the server includes means for acquiring video data to recognize the user's emotions in addition to sign language; means for extracting the characteristics of the sign language and emotions based on the user's facial expressions from the acquired video data; and means for analyzing the extracted characteristics of the sign language and emotions and converting them into the corresponding natural language while taking emotions into consideration. This enables rich, two-way communication, including emotions, between sign language and natural language.

[0179] Sign language is a visual language that involves hand and arm movements and facial expressions, used by people who are deaf or hard of hearing to communicate.

[0180] A "user" refers to a person who uses this system to communicate, engaging in dialogue through sign language or voice.

[0181] "Video data" refers to dynamic data that visually records sign language and facial expressions, acquired using imaging devices such as cameras.

[0182] "Emotions" refer to the psychological state that can be gleaned from a user's facial expressions and voice, and include information that encompasses nuances such as joy, anger, and sadness.

[0183] "Natural language" is a general term for spoken and written languages ​​that humans use on a daily basis, and it is a language that conveys meaning based on grammar.

[0184] "Audio output" refers to the process of playing back natural language generated by the server as audio, making it available for listening.

[0185] "Animation" is a type of video that visually represents movement by sequentially playing a series of still images, and is used to represent the movements of sign language.

[0186] This invention improves the quality of communication by incorporating an emotion recognition function into a system for mutual conversion between sign language and natural language. This system consists of a terminal that simultaneously captures the user's sign language and facial expressions, and a server that analyzes and converts that data.

[0187] The device has a built-in camera that captures the user's sign language movements and facial expressions in real time. This video data is compressed and sent to a server over the network. The device includes a high-resolution camera for proper video data capture and a high-performance chip for smooth processing.

[0188] The server uses advanced video processing software and machine learning algorithms to analyze the received video data. Specifically, it analyzes sign language movements and facial expressions to extract the content and emotions of the sign language. The server also uses an emotion engine to identify emotions such as smiles, anger, and sadness. This emotion information is taken into consideration during the natural language conversion process. Furthermore, a generative AI model is used to generate natural language text based on the extracted sign language and emotion information. This text is then converted into speech by a speech synthesis engine via a natural language processing (NLP) engine and sent to the terminal.

[0189] For example, if a user smiles while showing gratitude in sign language, the server will convert this into audio expressing the joyful emotion of "thank you." The terminal will then play this audio and relay it to a hearing person. An example of a prompt in this process would be: "Video has been captured of a user smiling and saying 'thank you' in sign language. Please translate this sign language appropriately and generate natural-sounding audio that includes the emotion of a smile."

[0190] The system also acquires voice input from sighted individuals and converts it to text through speech recognition on the server. It then generates and displays appropriate emotionally charged sign language animations to the user. This enables emotionally engaging two-way communication between visually impaired and sighted individuals.

[0191] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0192] Step 1:

[0193] The user performs a sign language and facial expression routine. Subsequently, the device's camera simultaneously captures the user's sign language movements and facial expressions. Video data is the input, and data containing hand and facial movement information is output. This data is appropriately compressed by a codec and sent to the server.

[0194] Step 2:

[0195] The server decodes the received compressed video data and uses machine learning algorithms to extract features from sign language movements and facial expressions. Here, video analysis software is used to extract meaningful hand and facial movements from the video data. This becomes the input, and sign language movement features and facial expression data are output.

[0196] Step 3:

[0197] The server inputs the extracted features into the emotion engine to recognize the user's emotional state. The emotion engine identifies emotions such as joy and anger based on changes in facial expressions, particularly eyebrow and mouth movements. The input consists of sign language gesture features and facial expression data, and the output is emotional information.

[0198] Step 4:

[0199] The server fuses the analyzed sign language features and emotional information, and uses a generative AI model to convert this into natural language text. This process generates appropriate sentences that retain the meaning and emotional nuances of the sign language. The input is sign language features and emotional information, and the output is natural language text.

[0200] Step 5:

[0201] The server inputs the generated natural language text into a speech synthesis engine and converts it into a human-readable speech format. Speech synthesis generates speech that reflects emotions based on the content of the text. The input is natural language text, and the output is synthesized speech data.

[0202] Step 6:

[0203] The device receives synthesized speech transmitted from the server and plays it back through its speaker. Here, audio data serves as input, and an emotionally charged audio output is produced for a hearing person. The device effectively conveys the content and emotions of sign language through audio playback.

[0204] Step 7:

[0205] The terminal captures voice input from a healthy individual via a microphone and sends that data to a server. The audio data becomes the input, and the audio data is output in a format that is then sent to the server.

[0206] Step 8:

[0207] The server analyzes the audio sent to it using a speech recognition engine and converts it into natural language text. The input is audio data, and the output is the converted text.

[0208] Step 9:

[0209] The server generates sign language animations that include appropriate emotional nuances based on natural language text. A generative AI model is used to create motion data from text. The input is natural language text, and the output is sign language animation data.

[0210] Step 10:

[0211] The terminal plays sign language animation data received from the server on the screen. This allows the user to visually understand messages from hearing individuals. The input is sign language animation data, and the output is the movements displayed on the screen.

[0212] (Application Example 2)

[0213] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0214] In modern times, accurately conveying emotional nuances and intentions is difficult in communication between sign language users and those who primarily use spoken language. This is especially true in caregiving settings, where understanding the emotions of the service user is crucial, and appropriate responses based on that understanding are required. In this context, there is a need to provide a bidirectional conversion system between natural language and sign language that takes emotions into consideration.

[0215] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0216] In this invention, the server includes means for acquiring information, means for extracting sign language features from the acquired information, means for analyzing the extracted features and recognizing emotions, means for converting the emotions into the corresponding natural language, and means for outputting the converted natural language data as speech. This enables two-way communication between sign language and natural language that takes emotions into account.

[0217] "Means of acquiring information" refers to devices and technologies for capturing data related to communication, such as sign language and voice.

[0218] "Methods for extracting the characteristics of sign language" refer to technologies that identify the characteristics of hand movements and facial expressions from acquired video data and extract them in an analyzable format.

[0219] "Means of recognizing emotions" refers to technologies that analyze and identify a user's emotions from sign language or voice data.

[0220] "Methods for converting into natural language while reflecting emotions" refers to technologies that convert sign language or speech into corresponding natural language text while taking emotions into consideration.

[0221] "Means for outputting natural language data as speech" refers to technology that converts text data into speech and provides it to the user in an audible format.

[0222] "Means for acquiring speech and converting it into natural language that takes emotions into account" refers to technology that recognizes speech data and converts it into natural language text that reflects the speaker's emotions.

[0223] "A means of outputting sign language motion data as animation" refers to a technology that visually represents data converted into sign language and presents it to the user as animation.

[0224] "Means for learning individual user emotional patterns" refers to a technology that accumulates each user's unique emotional patterns to improve the system's conversion accuracy.

[0225] The system for implementing this invention is designed to enable emotionally rich communication between sign language users and spoken language users in nursing care facilities. Specific embodiments of the system are described below.

[0226] System configuration:

[0227] The server has high processing power to receive and analyze sign language and speech data, and uses open-source image processing libraries (e.g., OpenCV, dlib) and emotion recognition APIs (e.g., Microsoft® Azure® Emotion API). It also utilizes AI services (e.g., Google Cloud Text-to-Speech API) for natural language processing and speech synthesis.

[0228] The terminals are portable devices such as smartphones and tablets that are equipped with cameras and microphones. This allows the terminals to directly acquire sign language and voice data and transmit it to the server in real time.

[0229] The users include both individuals who primarily use sign language and staff who use spoken language, and they communicate interactively through the system.

[0230] Example of operation:

[0231] When a user conveys "thank you" in sign language, the data, including their facial expressions, is sent from the device to the server. The server recognizes the emotion from the sign language data and facial expressions and converts it into natural language. Then, the message "thank you" is played back from the device's speaker in a cheerful voice that reflects the emotion. Conversely, if the message "that's wonderful" is conveyed verbally, it is presented to the user as an emotionally charged sign language animation.

[0232] Example prompts for generative AI models:

[0233] "If a user makes a smiling face and signs 'thank you,' what tone of voice will be generated?"

[0234] This invention aims to enrich communication in caregiving settings by enabling step-by-step information exchange between sign language and natural language.

[0235] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0236] Step 1:

[0237] The device uses its built-in camera to capture the user's sign language and facial expressions in real time. The frames captured as video data become the input. This data is then sent to the server for the next step.

[0238] Step 2:

[0239] The server receives the transmitted video data and uses an image processing library (e.g., OpenCV) to extract the characteristics of sign language from hand movements and facial features. During this process, specific motion data and facial expression data of the sign language are output.

[0240] Step 3:

[0241] The server uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to analyze the user's emotions from the extracted facial expression data. This process outputs information about the user's emotional state, which is then used in subsequent processing.

[0242] Step 4:

[0243] The server converts sign language motion data and user emotion information into natural language text. To reflect emotions, a generative AI model is used to adjust the nuances of the natural language. The resulting natural language text is then output.

[0244] Step 5:

[0245] The server converts natural language text into speech using the Google Cloud Text-to-Speech API. The generated speech data is output and sent to the device.

[0246] Step 6:

[0247] The terminal plays the audio data received from the server through its speaker. In this step, the user is provided with audio information that reflects emotions.

[0248] Step 7:

[0249] Conversely, the device uses its microphone to capture the voice of a healthy person and sends the audio data to the server. In this case, the input is audio data.

[0250] Step 8:

[0251] The server uses speech recognition technology to convert speech data into text data and then performs emotion recognition. The analysis results are output as natural language text and emotion information.

[0252] Step 9:

[0253] The server generates appropriate emotional sign language action data based on natural language text and emotional information. This results in the output of sign language animation data, which is then sent to the terminal.

[0254] Step 10:

[0255] The terminal plays the received sign language animation data on the screen, providing it in a visible format to the user communicating in sign language. This completes the entire process.

[0256] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0257] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0258] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0259] [Second Embodiment]

[0260] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0261] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0262] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0263] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0264] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0265] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0266] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0267] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0268] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0269] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0270] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0271] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0272] The real-time sign language translation system according to the present invention is a system that enables smooth communication between hearing individuals who do not understand sign language and hearing-impaired individuals. This system acquires sign language video in real time and converts it into speech or text to convey the intended message. It also has the function of converting the speech of hearing individuals into sign language and conveying it visually.

[0273] The device captures the user's sign language in real time via its camera. The video data is transmitted to the server with low latency. The server analyzes the received data and extracts the characteristics of the sign language. This identifies which sign language was shown and converts it into corresponding natural language text. The converted text is output as speech using speech synthesis technology and played back on the device.

[0274] Conversely, the device acquires the voice spoken by a hearing person using a microphone and sends that voice data to a server. The server converts the voice data into text using speech recognition technology, and then converts the text into appropriate sign language movements. These sign language movements are displayed as animations on the device, conveying the intended message to the hearing impaired.

[0275] As a concrete example, if a user (who is hearing impaired) expresses "Good morning" in sign language, the terminal sends the video to the server, which analyzes the sign language and converts it into the text "Good morning." This text is then output as audio and instantly transmitted to a hearing person.

[0276] This system goes beyond simple sign language-to-speech conversion; it has the ability to learn each user's individual sign language style and speech patterns, improving its conversion accuracy over time. This enables highly accurate translations, making it useful in a variety of communication situations.

[0277] The following describes the processing flow.

[0278] Step 1:

[0279] The device captures the user's sign language in real time via its camera. It acquires the video at a high frame rate and stores the visual information as digital data.

[0280] Step 2:

[0281] The terminal compresses and encodes the captured video data and transmits it to the server with low latency through the communication line.

[0282] Step 3:

[0283] The server extracts features such as the position of the hand, movement, and finger shape from the received video data. It identifies the necessary feature points using computer vision algorithms.

[0284] Step 4:

[0285] Based on the extracted feature points, the server uses a deep learning model to recognize sign language. It uses a pre-trained model to generate corresponding natural language text data.

[0286] Step 5:

[0287] The server uses natural language processing technology to generate synthetic voice from the obtained text data. Thereby, it creates an audio output as audio data.

[0288] Step 6:

[0289] The server transmits the generated audio data to the terminal.

[0290] Step 7:

[0291] The terminal plays back the received audio data through the speaker and conveys the content of the sign language to the healthy person as audio.

[0292] Step 8:

[0293] The terminal acquires the voice spoken by the healthy person through the microphone and transmits the audio data to the server.

[0294] Step 9:

[0295] The server converts the received audio data into text using speech recognition technology. By obtaining the text data, it prepares for the next processing step.

[0296] Step 10:

[0297] The server converts the obtained text data into sign language motion data. It then refers to a sign language database to generate specific sign language animations.

[0298] Step 11:

[0299] The server sends the generated sign language animation data to the terminal.

[0300] Step 12:

[0301] The device displays received sign language animations on its screen, visually conveying messages to people with hearing impairments.

[0302] (Example 1)

[0303] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0304] Facilitating smooth communication between deaf individuals who use sign language and hearing individuals who do not understand sign language is difficult. Conventional systems struggle to provide accurate real-time translation, thus creating a need for technology that efficiently performs bidirectional conversion between sign language and speech or text.

[0305] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0306] In this invention, the server includes a device for acquiring video information, a means for extracting sign language features from the acquired video information, and a mechanism for analyzing the extracted features and converting them into corresponding natural language. As a result, it becomes possible to convert sign language into natural language with high accuracy in real time and output it as voice.

[0307] "Video information" is dynamic visual data acquired to represent the movements of sign language.

[0308] "Sign language features" are distinguishable elements such as the shape, movement, and position of the hand in the sign language movements.

[0309] "Natural language" is the language used by humans in normal communication, referring to what is expressed as voice or text.

[0310] "Synthetic voice" is a technology that converts text information of natural language into a voice signal and reproduces it as mechanically generated voice.

[0311] "Voice information" is data obtained from voice and is used for performing voice recognition processing.

[0312] "Sign language movement information" is movement data that constitutes sign language animation generated based on voice or text.

[0313] "Animation" is a continuous visual representation used to visually represent sign language and its movements.

[0314] "Individual characteristics of users" refer to the movement and voice patterns unique to each individual user in sign language or speech.

[0315] "Learning device" is a processing device that continuously learns the characteristics of users' sign language and voice to improve conversion accuracy.

[0316] The real-time sign language translation system according to this invention aims to enable smooth communication between hearing-impaired individuals who use sign language and hearing individuals who do not understand sign language. Specific embodiments for carrying out this invention are described below.

[0317] The device uses a camera to capture the user's sign language movements in real time. The video information is processed by specialized equipment such as a Movidius Neural Compute Stick and transmitted to a server with low latency. The video information acquired here is high-quality visual data for accurately capturing sign language movements.

[0318] The server analyzes the received video information using a deep learning model. A CNN model utilizing TensorFlow is effective as the model, and this model functions as a mechanism to extract sign language features and convert them into corresponding natural language. In this process, the server identifies sign language features such as hand shape, movement, and position. The natural language converted based on the analysis is output as synthesized speech using speech synthesis technology such as the Google Text-to-Speech API.

[0319] Conversely, voice information spoken by a healthy person through the microphone on the device is also sent to the server. The server uses the Google Speech-to-Text API to convert this voice information into natural language text. This is then converted into sign language gesture information and visually displayed on the device using animation software such as Unity3D.

[0320] As a concrete example, if a user (who is hearing impaired) expresses "thank you" in sign language, the device sends the video to the server. The server analyzes the sign language, converts it into natural language text "thank you," and outputs it as synthesized speech to a hearing person.

[0321] One possible prompt to input into the generative AI model is: "Capture a video using a specific sign language action as an example, and explain how it is converted into speech."

[0322] This system also has a function that improves conversion accuracy by learning the user's individual sign language style and voice patterns over time, and is expected to be used in an even wider range of communication scenarios.

[0323] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0324] Step 1:

[0325] The device captures the user's sign language movements in real time via its camera. During this process, the device acquires video information and records the images at a high frame rate. The input is the user's sign language movements, and the output is raw video data.

[0326] Step 2:

[0327] The terminal compresses the acquired video information into the required format and sends it to the server. Here, the video is optimized to streamline data transfer and enable low-latency communication. The input is raw video data, and the output is compressed video data.

[0328] Step 3:

[0329] The server receives the transmitted compressed video data and extracts sign language features using a deep learning model. Specifically, a model using TensorFlow analyzes the shape and movement of the hands and recognizes feature points based on that analysis. The input is compressed video data, and the output is sign language feature information.

[0330] Step 4:

[0331] The server utilizes a sign language-to-text mapping database to convert extracted sign language feature information into natural language. This analysis converts the intent conveyed by the sign language into natural language sentences. The input is sign language feature information, and the output is natural language text data.

[0332] Step 5:

[0333] The server converts the translated natural language text into speech data using speech synthesis technology. The Google Text-to-Speech API is used for the speech generation process. The input is natural language text data, and the output is synthesized speech.

[0334] Step 6:

[0335] The device receives synthesized speech transmitted from the server and plays the speech through its built-in speaker. This allows hearing individuals to understand the meaning of sign language as spoken language. The input is synthesized speech data, and the output is spoken audio.

[0336] Step 7:

[0337] The device captures voice information spoken by a healthy individual using a microphone. The acquired voice information is recorded in clear quality. The input is the voice of a healthy individual, and the output is voice data.

[0338] Step 8:

[0339] The terminal optimizes and compresses the recorded audio data before sending it to the server. It then transmits it with low latency. The input is audio data, and the output is compressed audio data.

[0340] Step 9:

[0341] The server converts the received audio data into natural language text using speech recognition technology. The Google Speech-to-Text API analyzes the content of the audio and converts it into text format. The input is compressed audio data, and the output is text data.

[0342] Step 10:

[0343] The server converts text data into sign language motion information. In this process, it uses an animation engine to convert the text into motion data in order to generate corresponding sign language animations. The input is text data, and the output is sign language motion information.

[0344] Step 11:

[0345] The terminal displays sign language gesture information received from the server as an animation on its screen. This allows the user (a person with hearing impairment) to receive messages visually. The input is sign language gesture information, and the output is a sign language animation.

[0346] (Application Example 1)

[0347] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0348] In modern society, the lack of effective means of smooth communication between the hearing impaired and hearing individuals remains a significant challenge. For the hearing impaired, whose primary means of communication is sign language, communication with hearing individuals is often difficult in many situations in daily life. Therefore, there is a need for real-time conversion between sign language and spoken language to achieve smooth and accurate communication. Furthermore, providing conversion accuracy optimized for each individual user is also crucial.

[0349] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0350] In this invention, the server includes means for acquiring video data for recognizing sign language, means for acquiring audio data and converting it into natural language, and means for learning user characteristics using feedback information and improving conversion accuracy. This enables real-time and highly accurate communication between hearing-impaired and hearing individuals.

[0351] "Means for acquiring video data for recognizing sign language" refers to methods for recording sign language movements in real time using a camera mounted on a terminal or device.

[0352] "Methods for extracting the characteristics of sign language" refer to algorithms that identify specific hand shapes and movement patterns from acquired sign language video data and analyze the meaning of the sign language.

[0353] "Means for converting to corresponding natural language" refers to methods for converting extracted sign language features into words and treating them as text data with concrete meaning.

[0354] "Means for converting converted natural language data into speech and outputting it via a speech output device" refers to a technology for converting generated text data into sound waves using speech synthesis technology and then playing them back through an acoustic device.

[0355] "Means for acquiring audio data and converting it into natural language" refers to speech recognition technology that analyzes input audio data and expresses its content as text.

[0356] "Means for converting natural language into sign language motion data" refers to algorithms for converting natural language text information into sequences of actions that can be communicated in sign language.

[0357] "Means for displaying sign language motion data on a visual display device" refers to a display technology that animates generated sign language motions and presents them visually.

[0358] "Methods for learning user characteristics using feedback information and improving conversion accuracy" refers to a function that analyzes the user's usage history and feedback data, optimizes the conversion algorithm, and achieves highly accurate translations tailored to individual users.

[0359] This invention is a system for facilitating communication using sign language. The system mainly consists of a terminal worn by the user and a server connected to it.

[0360] The devices are smart glasses or other mobile devices. They have built-in cameras and microphones that capture the user's sign language and voice, respectively. The sign language video data captured by the camera is transmitted to a server with low latency. The server analyzes the acquired video using machine learning models such as TensorFlow Lite to extract sign language features. This analysis identifies the meaning of the sign language movements. The identified sign language features are converted into natural language text and then spoken using speech synthesis technology. This allows the information to be conveyed to hearing individuals in both text and audio formats.

[0361] Conversely, voice input from able-bodied individuals is captured by the device and sent to the server. The server uses the Google Cloud Speech-to-Text API to convert this voice data into natural language text. Then, a generative AI model is used to convert the text into sign language motion data. This sign language motion is then animated using software such as Unity and displayed on the device's screen.

[0362] Furthermore, the system learns user characteristics and improves translation accuracy by receiving feedback as needed. This enables the achievement of highly accurate translations optimized for individual users.

[0363] For example, if a user (who is hearing impaired) uses sign language to mean "turn up the TV volume," the server analyzes this sign language and converts it into the natural language phrase "Please turn up the TV volume." Conversely, if a hearing person says "Good morning," the system animates this speech as sign language and conveys it to the user. An example of a prompt message to the generative AI model would be: "Analyze the sign language actions captured by the user via the camera and output their intent as spoken language. Also, display the spoken language as a sign language animation on the screen."

[0364] This system makes it possible for both people with hearing impairments and those without hearing impairments to communicate without difficulty.

[0365] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0366] Step 1:

[0367] The device captures the user's sign language expressions with its camera and temporarily stores the video data. Next, it transmits this video data to the server with low latency. The input is the user's sign language actions, and the output is the transfer of video data to the server.

[0368] Step 2:

[0369] The server uses TensorFlow Lite to extract sign language features from the received video data. The input is captured video data of sign language, and the output is a sign language feature vector. This vector is used to analyze the content of the sign language and convert it into natural language text.

[0370] Step 3:

[0371] The server converts the generated natural language text into speech using speech synthesis technology (e.g., a text-to-speech engine). The input is natural language text, and the output is audio data. This audio data is sent to the terminal and played back via an audio output device.

[0372] Step 4:

[0373] In the reverse process, the terminal acquires voice from a healthy person using a microphone and sends that voice data to the server. The input is the voice of the healthy person, and the output is the transfer of voice data to the server.

[0374] Step 5:

[0375] The server uses the Google Cloud Speech-to-Text API to convert speech data into natural language text. The input is speech data from a healthy individual, and the output is natural language text. Based on this text, a generative AI model is used to convert the text into sign language gesture data.

[0376] Step 6:

[0377] The server animates the converted sign language motion data using software such as Unity, and sends this animated data to the terminal. The input is the sign language motion data, and the output is the transmission of the animated sign language data to the terminal.

[0378] Step 7:

[0379] The terminal displays animated sign language to the user. The input is animation data from the server, and the output is sign language information visually communicated to the user.

[0380] Step 8:

[0381] Based on user and healthy user feedback, the server updates the conversion model and learns to achieve optimized conversion accuracy for each user. The input is the feedback data and existing model parameters, and the output is the optimized conversion model.

[0382] This process enables real-time translation between sign language and spoken language, thereby lowering communication barriers.

[0383] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0384] This invention aims to achieve richer communication by adding a function to recognize user emotions to a system that performs mutual conversion between sign language and natural language. This system is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice during the process of extracting the characteristics of sign language, and can convert them into natural language text while taking those emotions into consideration.

[0385] The device captures the user's sign language and facial expressions using its camera. The video data is sent to a server, where characteristic hand and facial movements are analyzed. In addition to the content of the sign language, the server uses an emotion engine to recognize the user's emotions, identifying expressions such as smiles, anger, and sadness. This extracts the nuances of emotion conveyed in the spoken content.

[0386] Based on the analyzed data, the server generates natural language text that reflects emotions and outputs it as synthesized speech. The terminal plays the received audio through its speaker, conveying the content of the sign language and the emotions behind it as audio information to a hearing person. For example, if a user signs "hello" with a smile, the audio output will say "hello" in a cheerful tone.

[0387] Furthermore, when the device acquires the voice of a healthy person, the server converts the voice into text and generates appropriate sign language actions with corresponding emotions from that text. The generated sign language data is sent to the device as an animation and displayed to the user. At this time, based on the emotion data, for example, a word pronounced with emotion, such as "wonderful," will be reproduced with a sign language animation that conveys the appropriate emotion.

[0388] By incorporating emotion recognition in this way, it becomes possible to enable more emotionally rich two-way communication between natural language and sign language than before. This feature will be particularly useful for users in situations where complex emotions need to be conveyed, or in informal communication.

[0389] The following describes the processing flow.

[0390] Step 1:

[0391] The device uses a camera to simultaneously capture the user's sign language movements and facial expressions. High-resolution video data is recorded in real time and transmitted to a server via the network.

[0392] Step 2:

[0393] The server extracts features corresponding to sign language movements from the received video data. In addition to hand position and movement, it analyzes facial expression data to capture emotional elements along with the sign language.

[0394] Step 3:

[0395] The server uses an emotion engine to recognize the user's emotions from the extracted facial expression data. It identifies the type of emotion (e.g., joy, sadness, surprise, etc.) and analyzes this information in conjunction with the characteristics of sign language.

[0396] Step 4:

[0397] The server generates corresponding natural language text based on the characteristics and emotional information of the sign language. The emotional information is reflected in the text to ensure that the expression includes emotional nuances.

[0398] Step 5:

[0399] The server converts the generated text data into synthesized speech. Based on emotional information, it adjusts the tone and speed of the speech to create speech data that conveys emotion.

[0400] Step 6:

[0401] The server sends synthesized speech data to the terminal, and the terminal plays the received speech to the user through its speaker. A healthy person receives the emotionally charged sign language content as speech.

[0402] Step 7:

[0403] The device captures speech uttered by a healthy individual using its microphone and sends that audio data to a server.

[0404] Step 8:

[0405] The server uses speech recognition technology to convert the audio data into text, analyzes its content, and estimates the emotions conveyed.

[0406] Step 9:

[0407] The server generates sign language action data with appropriate emotional expressions based on estimated emotional information. It then creates sign language animations that reflect emotions by referring to a sign language database.

[0408] Step 10:

[0409] The server sends the generated sign language animation data to the terminal.

[0410] Step 11:

[0411] The device displays sign language animations on its screen, conveying emotions through sign language in a way that is visible to the hearing impaired. As a result, users can visually understand emotions along with the audio content.

[0412] (Example 2)

[0413] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0414] Conventional sign language translation systems failed to consider the user's emotions when converting sign language movements into natural language, resulting in a lack of emotional transmission in communication. Furthermore, even when converting spoken language from hearing individuals into sign language, generating sign language movements that reflected appropriate emotions was difficult. This made it challenging to accurately convey intentions, especially in conversations where emotional nuances are crucial.

[0415] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0416] In this invention, the server includes means for acquiring video data to recognize the user's emotions in addition to sign language; means for extracting the characteristics of the sign language and emotions based on the user's facial expressions from the acquired video data; and means for analyzing the extracted characteristics of the sign language and emotions and converting them into the corresponding natural language while taking emotions into consideration. This enables rich, two-way communication, including emotions, between sign language and natural language.

[0417] Sign language is a visual language that involves hand and arm movements and facial expressions, used by people who are deaf or hard of hearing to communicate.

[0418] A "user" refers to a person who uses this system to communicate, engaging in dialogue through sign language or voice.

[0419] "Video data" refers to dynamic data that visually records sign language and facial expressions, acquired using imaging devices such as cameras.

[0420] "Emotions" refer to the psychological state that can be gleaned from a user's facial expressions and voice, and include information that encompasses nuances such as joy, anger, and sadness.

[0421] "Natural language" is a general term for spoken and written languages ​​that humans use on a daily basis, and it is a language that conveys meaning based on grammar.

[0422] "Audio output" refers to the process of playing back natural language generated by the server as audio, making it available for listening.

[0423] "Animation" is a type of video that visually represents movement by sequentially playing a series of still images, and is used to represent the movements of sign language.

[0424] This invention improves the quality of communication by incorporating an emotion recognition function into a system for mutual conversion between sign language and natural language. This system consists of a terminal that simultaneously captures the user's sign language and facial expressions, and a server that analyzes and converts that data.

[0425] The device has a built-in camera that captures the user's sign language movements and facial expressions in real time. This video data is compressed and sent to a server over the network. The device includes a high-resolution camera for proper video data capture and a high-performance chip for smooth processing.

[0426] The server uses advanced video processing software and machine learning algorithms to analyze the received video data. Specifically, it analyzes sign language movements and facial expressions to extract the content and emotions of the sign language. The server also uses an emotion engine to identify emotions such as smiles, anger, and sadness. This emotion information is taken into consideration during the natural language conversion process. Furthermore, a generative AI model is used to generate natural language text based on the extracted sign language and emotion information. This text is then converted into speech by a speech synthesis engine via a natural language processing (NLP) engine and sent to the terminal.

[0427] For example, if a user smiles while showing gratitude in sign language, the server will convert this into audio expressing the joyful emotion of "thank you." The terminal will then play this audio and relay it to a hearing person. An example of a prompt in this process would be: "Video has been captured of a user smiling and saying 'thank you' in sign language. Please translate this sign language appropriately and generate natural-sounding audio that includes the emotion of a smile."

[0428] The system also acquires voice input from sighted individuals and converts it to text through speech recognition on the server. It then generates and displays appropriate emotionally charged sign language animations to the user. This enables emotionally engaging two-way communication between visually impaired and sighted individuals.

[0429] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0430] Step 1:

[0431] The user performs a sign language and facial expression routine. Subsequently, the device's camera simultaneously captures the user's sign language movements and facial expressions. Video data is the input, and data containing hand and facial movement information is output. This data is appropriately compressed by a codec and sent to the server.

[0432] Step 2:

[0433] The server decodes the received compressed video data and uses machine learning algorithms to extract features from sign language movements and facial expressions. Here, video analysis software is used to extract meaningful hand and facial movements from the video data. This becomes the input, and sign language movement features and facial expression data are output.

[0434] Step 3:

[0435] The server inputs the extracted features into the emotion engine to recognize the user's emotional state. The emotion engine identifies emotions such as joy and anger based on changes in facial expressions, particularly eyebrow and mouth movements. The input consists of sign language gesture features and facial expression data, and the output is emotional information.

[0436] Step 4:

[0437] The server fuses the analyzed sign language features and emotional information, and uses a generative AI model to convert this into natural language text. This process generates appropriate sentences that retain the meaning and emotional nuances of the sign language. The input is sign language features and emotional information, and the output is natural language text.

[0438] Step 5:

[0439] The server inputs the generated natural language text into a speech synthesis engine and converts it into a human-readable speech format. Speech synthesis generates speech that reflects emotions based on the content of the text. The input is natural language text, and the output is synthesized speech data.

[0440] Step 6:

[0441] The device receives synthesized speech transmitted from the server and plays it back through its speaker. Here, audio data serves as input, and an emotionally charged audio output is produced for a hearing person. The device effectively conveys the content and emotions of sign language through audio playback.

[0442] Step 7:

[0443] The terminal captures voice input from a healthy individual via a microphone and sends that data to a server. The audio data becomes the input, and the audio data is output in a format that is then sent to the server.

[0444] Step 8:

[0445] The server analyzes the audio sent to it using a speech recognition engine and converts it into natural language text. The input is audio data, and the output is the converted text.

[0446] Step 9:

[0447] The server generates sign language animations that include appropriate emotional nuances based on natural language text. A generative AI model is used to create motion data from text. The input is natural language text, and the output is sign language animation data.

[0448] Step 10:

[0449] The terminal plays sign language animation data received from the server on the screen. This allows the user to visually understand messages from hearing individuals. The input is sign language animation data, and the output is the movements displayed on the screen.

[0450] (Application Example 2)

[0451] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0452] In modern times, accurately conveying emotional nuances and intentions is difficult in communication between sign language users and those who primarily use spoken language. This is especially true in caregiving settings, where understanding the emotions of the service user is crucial, and appropriate responses based on that understanding are required. In this context, there is a need to provide a bidirectional conversion system between natural language and sign language that takes emotions into consideration.

[0453] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0454] In this invention, the server includes means for acquiring information, means for extracting sign language features from the acquired information, means for analyzing the extracted features and recognizing emotions, means for converting the emotions into the corresponding natural language, and means for outputting the converted natural language data as speech. This enables two-way communication between sign language and natural language that takes emotions into account.

[0455] "Means of acquiring information" refers to devices and technologies for capturing data related to communication, such as sign language and voice.

[0456] "Methods for extracting the characteristics of sign language" refer to technologies that identify the characteristics of hand movements and facial expressions from acquired video data and extract them in an analyzable format.

[0457] "Means of recognizing emotions" refers to technologies that analyze and identify a user's emotions from sign language or voice data.

[0458] "Methods for converting into natural language while reflecting emotions" refers to technologies that convert sign language or speech into corresponding natural language text while taking emotions into consideration.

[0459] "Means for outputting natural language data as speech" refers to technology that converts text data into speech and provides it to the user in an audible format.

[0460] "Means for acquiring speech and converting it into natural language that takes emotions into account" refers to technology that recognizes speech data and converts it into natural language text that reflects the speaker's emotions.

[0461] "A means of outputting sign language motion data as animation" refers to a technology that visually represents data converted into sign language and presents it to the user as animation.

[0462] "Means for learning individual user emotional patterns" refers to a technology that accumulates each user's unique emotional patterns to improve the system's conversion accuracy.

[0463] The system for implementing this invention is designed to enable emotionally rich communication between sign language users and spoken language users in nursing care facilities. Specific embodiments of the system are described below.

[0464] System configuration:

[0465] The server has high processing power to receive and analyze sign language and speech data, and uses open-source image processing libraries (e.g., OpenCV, dlib) and emotion recognition APIs (e.g., Microsoft Azure Emotion API). It also utilizes AI services (e.g., Google Cloud Text-to-Speech API) for natural language processing and speech synthesis.

[0466] The terminals are portable devices such as smartphones and tablets that are equipped with cameras and microphones. This allows the terminals to directly acquire sign language and voice data and transmit it to the server in real time.

[0467] The users include both individuals who primarily use sign language and staff who use spoken language, and they communicate interactively through the system.

[0468] Example of operation:

[0469] When a user conveys "thank you" in sign language, the data, including their facial expressions, is sent from the device to the server. The server recognizes the emotion from the sign language data and facial expressions and converts it into natural language. Then, the message "thank you" is played back from the device's speaker in a cheerful voice that reflects the emotion. Conversely, if the message "that's wonderful" is conveyed verbally, it is presented to the user as an emotionally charged sign language animation.

[0470] Example prompts for generative AI models:

[0471] "If a user makes a smiling face and signs 'thank you,' what tone of voice will be generated?"

[0472] This invention aims to enrich communication in caregiving settings by enabling step-by-step information exchange between sign language and natural language.

[0473] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0474] Step 1:

[0475] The device uses its built-in camera to capture the user's sign language and facial expressions in real time. The frames captured as video data become the input. This data is then sent to the server for the next step.

[0476] Step 2:

[0477] The server receives the transmitted video data and uses an image processing library (e.g., OpenCV) to extract the characteristics of sign language from hand movements and facial features. During this process, specific motion data and facial expression data of the sign language are output.

[0478] Step 3:

[0479] The server uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to analyze the user's emotions from the extracted facial expression data. This process outputs information about the user's emotional state, which is then used in subsequent processing.

[0480] Step 4:

[0481] The server converts sign language motion data and user emotion information into natural language text. To reflect emotions, a generative AI model is used to adjust the nuances of the natural language. The resulting natural language text is then output.

[0482] Step 5:

[0483] The server converts natural language text into speech using the Google Cloud Text-to-Speech API. The generated speech data is output and sent to the device.

[0484] Step 6:

[0485] The terminal plays the audio data received from the server through its speaker. In this step, the user is provided with audio information that reflects emotions.

[0486] Step 7:

[0487] Conversely, the device uses its microphone to capture the voice of a healthy person and sends the audio data to the server. In this case, the input is audio data.

[0488] Step 8:

[0489] The server uses speech recognition technology to convert speech data into text data and then performs emotion recognition. The analysis results are output as natural language text and emotion information.

[0490] Step 9:

[0491] The server generates appropriate emotional sign language action data based on natural language text and emotional information. This results in the output of sign language animation data, which is then sent to the terminal.

[0492] Step 10:

[0493] The terminal plays the received sign language animation data on the screen, providing it in a visible format to the user communicating in sign language. This completes the entire process.

[0494] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0495] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0496] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0497] [Third Embodiment]

[0498] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0499] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0500] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0501] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0502] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0503] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0504] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0505] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0506] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0507] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0508] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0509] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0510] The real-time sign language translation system according to the present invention is a system that enables smooth communication between hearing individuals who do not understand sign language and hearing-impaired individuals. This system acquires sign language video in real time and converts it into speech or text to convey the intended message. It also has the function of converting the speech of hearing individuals into sign language and conveying it visually.

[0511] The device captures the user's sign language in real time via its camera. The video data is transmitted to the server with low latency. The server analyzes the received data and extracts the characteristics of the sign language. This identifies which sign language was shown and converts it into corresponding natural language text. The converted text is output as speech using speech synthesis technology and played back on the device.

[0512] Conversely, the device acquires the voice spoken by a hearing person using a microphone and sends that voice data to a server. The server converts the voice data into text using speech recognition technology, and then converts the text into appropriate sign language movements. These sign language movements are displayed as animations on the device, conveying the intended message to the hearing impaired.

[0513] As a concrete example, if a user (who is hearing impaired) expresses "Good morning" in sign language, the terminal sends the video to the server, which analyzes the sign language and converts it into the text "Good morning." This text is then output as audio and instantly transmitted to a hearing person.

[0514] This system goes beyond simple sign language-to-speech conversion; it has the ability to learn each user's individual sign language style and speech patterns, improving its conversion accuracy over time. This enables highly accurate translations, making it useful in a variety of communication situations.

[0515] The following describes the processing flow.

[0516] Step 1:

[0517] The device captures the user's sign language in real time via its camera. It acquires the video at a high frame rate and stores the visual information as digital data.

[0518] Step 2:

[0519] The terminal compresses and encodes the captured video data and transmits it to the server via the communication line with low latency.

[0520] Step 3:

[0521] The server extracts features such as hand position, movement, and finger shape from the received video data. It uses computer vision algorithms to identify the necessary feature points.

[0522] Step 4:

[0523] The server uses a deep learning model to recognize sign language based on the extracted feature points. Using the trained model, it generates corresponding natural language text data.

[0524] Step 5:

[0525] The server uses natural language processing technology to generate synthesized speech from the obtained text data. This creates audio output as audio data.

[0526] Step 6:

[0527] The server sends the generated audio data to the terminal.

[0528] Step 7:

[0529] The device plays the received audio data through its speaker, conveying the content of the sign language to a hearing person as audio.

[0530] Step 8:

[0531] The device acquires the voice spoken by a healthy person through its microphone and sends the voice data to the server.

[0532] Step 9:

[0533] The server converts the received audio data into text using speech recognition technology. By obtaining the text data, it prepares for the next processing step.

[0534] Step 10:

[0535] The server converts the obtained text data into sign language motion data. It then refers to a sign language database to generate specific sign language animations.

[0536] Step 11:

[0537] The server sends the generated sign language animation data to the terminal.

[0538] Step 12:

[0539] The device displays received sign language animations on its screen, visually conveying messages to people with hearing impairments.

[0540] (Example 1)

[0541] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0542] Facilitating smooth communication between deaf individuals who use sign language and hearing individuals who do not understand sign language is difficult. Conventional systems struggle to provide accurate real-time translation, thus creating a need for technology that efficiently performs bidirectional conversion between sign language and speech or text.

[0543] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0544] In this invention, the server includes a device for acquiring video information, means for extracting sign language features from the acquired video information, and a mechanism for analyzing the extracted features and converting them into corresponding natural language. This makes it possible to convert sign language into natural language in real time with high accuracy and output it as speech.

[0545] "Visual information" refers to dynamic visual data acquired to represent sign language movements.

[0546] "Characteristics of sign language" refer to identifiable elements such as the shape, movement, and position of the hands in sign language actions.

[0547] "Natural language" refers to the language that humans use for normal communication, and which can be expressed as spoken or written language.

[0548] "Synthesized speech" is a technology that converts natural language text information into audio signals and plays them back using mechanically generated speech.

[0549] "Audio information" refers to data obtained from speech and is used for speech recognition processing.

[0550] "Sign language motion information" refers to the motion data that makes up sign language animations generated based on audio and text.

[0551] "Animation" refers to a series of visual representations used to visually represent sign language and its movements.

[0552] "Individual user characteristics" refers to the unique actions and vocal patterns of each user in sign language and speech.

[0553] A "learning device" is a processing device that continuously learns the characteristics of the user's sign language and speech to improve conversion accuracy.

[0554] The real-time sign language translation system according to this invention aims to enable smooth communication between hearing-impaired individuals who use sign language and hearing individuals who do not understand sign language. Specific embodiments for carrying out this invention are described below.

[0555] The device uses a camera to capture the user's sign language movements in real time. The video information is processed by specialized equipment such as a Movidius Neural Compute Stick and transmitted to a server with low latency. The video information acquired here is high-quality visual data for accurately capturing sign language movements.

[0556] The server analyzes the received video information using a deep learning model. A CNN model utilizing TensorFlow is effective as the model, and this model functions as a mechanism to extract sign language features and convert them into corresponding natural language. In this process, the server identifies sign language features such as hand shape, movement, and position. The natural language converted based on the analysis is output as synthesized speech using speech synthesis technology such as the Google Text-to-Speech API.

[0557] Conversely, voice information spoken by a healthy person through the microphone on the device is also sent to the server. The server uses the Google Speech-to-Text API to convert this voice information into natural language text. This is then converted into sign language gesture information and visually displayed on the device using animation software such as Unity3D.

[0558] As a concrete example, if a user (who is hearing impaired) expresses "thank you" in sign language, the device sends the video to the server. The server analyzes the sign language, converts it into natural language text "thank you," and outputs it as synthesized speech to a hearing person.

[0559] One possible prompt to input into the generative AI model is: "Capture a video using a specific sign language action as an example, and explain how it is converted into speech."

[0560] This system also has a function that improves conversion accuracy by learning the user's individual sign language style and voice patterns over time, and is expected to be used in an even wider range of communication scenarios.

[0561] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0562] Step 1:

[0563] The device captures the user's sign language movements in real time via its camera. During this process, the device acquires video information and records the images at a high frame rate. The input is the user's sign language movements, and the output is raw video data.

[0564] Step 2:

[0565] The terminal compresses the acquired video information into the required format and sends it to the server. Here, the video is optimized to streamline data transfer and enable low-latency communication. The input is raw video data, and the output is compressed video data.

[0566] Step 3:

[0567] The server receives the transmitted compressed video data and extracts sign language features using a deep learning model. Specifically, a model using TensorFlow analyzes the shape and movement of the hands and recognizes feature points based on that analysis. The input is compressed video data, and the output is sign language feature information.

[0568] Step 4:

[0569] The server utilizes a sign language-to-text mapping database to convert extracted sign language feature information into natural language. This analysis converts the intent conveyed by the sign language into natural language sentences. The input is sign language feature information, and the output is natural language text data.

[0570] Step 5:

[0571] The server converts the translated natural language text into speech data using speech synthesis technology. The Google Text-to-Speech API is used for the speech generation process. The input is natural language text data, and the output is synthesized speech.

[0572] Step 6:

[0573] The device receives synthesized speech transmitted from the server and plays the speech through its built-in speaker. This allows hearing individuals to understand the meaning of sign language as spoken language. The input is synthesized speech data, and the output is spoken audio.

[0574] Step 7:

[0575] The device captures voice information spoken by a healthy individual using a microphone. The acquired voice information is recorded in clear quality. The input is the voice of a healthy individual, and the output is voice data.

[0576] Step 8:

[0577] The terminal optimizes and compresses the recorded audio data before sending it to the server. It then transmits it with low latency. The input is audio data, and the output is compressed audio data.

[0578] Step 9:

[0579] The server converts the received audio data into natural language text using speech recognition technology. The Google Speech-to-Text API analyzes the content of the audio and converts it into text format. The input is compressed audio data, and the output is text data.

[0580] Step 10:

[0581] The server converts text data into sign language motion information. In this process, it uses an animation engine to convert the text into motion data in order to generate corresponding sign language animations. The input is text data, and the output is sign language motion information.

[0582] Step 11:

[0583] The terminal displays sign language gesture information received from the server as an animation on its screen. This allows the user (a person with hearing impairment) to receive messages visually. The input is sign language gesture information, and the output is a sign language animation.

[0584] (Application Example 1)

[0585] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0586] In modern society, the lack of effective means of smooth communication between the hearing impaired and hearing individuals remains a significant challenge. For the hearing impaired, whose primary means of communication is sign language, communication with hearing individuals is often difficult in many situations in daily life. Therefore, there is a need for real-time conversion between sign language and spoken language to achieve smooth and accurate communication. Furthermore, providing conversion accuracy optimized for each individual user is also crucial.

[0587] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0588] In this invention, the server includes means for acquiring video data for recognizing sign language, means for acquiring audio data and converting it into natural language, and means for learning user characteristics using feedback information and improving conversion accuracy. This enables real-time and highly accurate communication between hearing-impaired and hearing individuals.

[0589] "Means for acquiring video data for recognizing sign language" refers to methods for recording sign language movements in real time using a camera mounted on a terminal or device.

[0590] "Methods for extracting the characteristics of sign language" refer to algorithms that identify specific hand shapes and movement patterns from acquired sign language video data and analyze the meaning of the sign language.

[0591] "Means for converting to corresponding natural language" refers to methods for converting extracted sign language features into words and treating them as text data with concrete meaning.

[0592] "Means for converting converted natural language data into speech and outputting it via a speech output device" refers to a technology for converting generated text data into sound waves using speech synthesis technology and then playing them back through an acoustic device.

[0593] "Means for acquiring audio data and converting it into natural language" refers to speech recognition technology that analyzes input audio data and expresses its content as text.

[0594] "Means for converting natural language into sign language motion data" refers to algorithms for converting natural language text information into sequences of actions that can be communicated in sign language.

[0595] "Means for displaying sign language motion data on a visual display device" refers to a display technology that animates generated sign language motions and presents them visually.

[0596] "Methods for learning user characteristics using feedback information and improving conversion accuracy" refers to a function that analyzes the user's usage history and feedback data, optimizes the conversion algorithm, and achieves highly accurate translations tailored to individual users.

[0597] This invention is a system for facilitating communication using sign language. The system mainly consists of a terminal worn by the user and a server connected to it.

[0598] The devices are smart glasses or other mobile devices. They have built-in cameras and microphones that capture the user's sign language and voice, respectively. The sign language video data captured by the camera is transmitted to a server with low latency. The server analyzes the acquired video using machine learning models such as TensorFlow Lite to extract sign language features. This analysis identifies the meaning of the sign language movements. The identified sign language features are converted into natural language text and then spoken using speech synthesis technology. This allows the information to be conveyed to hearing individuals in both text and audio formats.

[0599] Conversely, voice input from able-bodied individuals is captured by the device and sent to the server. The server uses the Google Cloud Speech-to-Text API to convert this voice data into natural language text. Then, a generative AI model is used to convert the text into sign language motion data. This sign language motion is then animated using software such as Unity and displayed on the device's screen.

[0600] Furthermore, the system learns user characteristics and improves translation accuracy by receiving feedback as needed. This enables the achievement of highly accurate translations optimized for individual users.

[0601] For example, if a user (who is hearing impaired) uses sign language to mean "turn up the TV volume," the server analyzes this sign language and converts it into the natural language phrase "Please turn up the TV volume." Conversely, if a hearing person says "Good morning," the system animates this speech as sign language and conveys it to the user. An example of a prompt message to the generative AI model would be: "Analyze the sign language actions captured by the user via the camera and output their intent as spoken language. Also, display the spoken language as a sign language animation on the screen."

[0602] This system makes it possible for both people with hearing impairments and those without hearing impairments to communicate without difficulty.

[0603] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0604] Step 1:

[0605] The device captures the user's sign language expressions with its camera and temporarily stores the video data. Next, it transmits this video data to the server with low latency. The input is the user's sign language actions, and the output is the transfer of video data to the server.

[0606] Step 2:

[0607] The server uses TensorFlow Lite to extract sign language features from the received video data. The input is captured video data of sign language, and the output is a sign language feature vector. This vector is used to analyze the content of the sign language and convert it into natural language text.

[0608] Step 3:

[0609] The server converts the generated natural language text into speech using speech synthesis technology (e.g., a text-to-speech engine). The input is natural language text, and the output is audio data. This audio data is sent to the terminal and played back via an audio output device.

[0610] Step 4:

[0611] In the reverse process, the terminal acquires voice from a healthy person using a microphone and sends that voice data to the server. The input is the voice of the healthy person, and the output is the transfer of voice data to the server.

[0612] Step 5:

[0613] The server uses the Google Cloud Speech-to-Text API to convert speech data into natural language text. The input is speech data from a healthy individual, and the output is natural language text. Based on this text, a generative AI model is used to convert the text into sign language gesture data.

[0614] Step 6:

[0615] The server animates the converted sign language motion data using software such as Unity, and sends this animated data to the terminal. The input is the sign language motion data, and the output is the transmission of the animated sign language data to the terminal.

[0616] Step 7:

[0617] The terminal displays animated sign language to the user. The input is animation data from the server, and the output is sign language information visually communicated to the user.

[0618] Step 8:

[0619] Based on user and healthy user feedback, the server updates the conversion model and learns to achieve optimized conversion accuracy for each user. The input is the feedback data and existing model parameters, and the output is the optimized conversion model.

[0620] This process enables real-time translation between sign language and spoken language, thereby lowering communication barriers.

[0621] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0622] This invention aims to achieve richer communication by adding a function to recognize user emotions to a system that performs mutual conversion between sign language and natural language. This system is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice during the process of extracting the characteristics of sign language, and can convert them into natural language text while taking those emotions into consideration.

[0623] The device captures the user's sign language and facial expressions using its camera. The video data is sent to a server, where characteristic hand and facial movements are analyzed. In addition to the content of the sign language, the server uses an emotion engine to recognize the user's emotions, identifying expressions such as smiles, anger, and sadness. This extracts the nuances of emotion conveyed in the spoken content.

[0624] Based on the analyzed data, the server generates natural language text that reflects emotions and outputs it as synthesized speech. The terminal plays the received audio through its speaker, conveying the content of the sign language and the emotions behind it as audio information to a hearing person. For example, if a user signs "hello" with a smile, the audio output will say "hello" in a cheerful tone.

[0625] Furthermore, when the device acquires the voice of a healthy person, the server converts the voice into text and generates appropriate sign language actions with corresponding emotions from that text. The generated sign language data is sent to the device as an animation and displayed to the user. At this time, based on the emotion data, for example, a word pronounced with emotion, such as "wonderful," will be reproduced with a sign language animation that conveys the appropriate emotion.

[0626] By incorporating emotion recognition in this way, it becomes possible to enable more emotionally rich two-way communication between natural language and sign language than before. This feature will be particularly useful for users in situations where complex emotions need to be conveyed, or in informal communication.

[0627] The following describes the processing flow.

[0628] Step 1:

[0629] The device uses a camera to simultaneously capture the user's sign language movements and facial expressions. High-resolution video data is recorded in real time and transmitted to a server via the network.

[0630] Step 2:

[0631] The server extracts features corresponding to sign language movements from the received video data. In addition to hand position and movement, it analyzes facial expression data to capture emotional elements along with the sign language.

[0632] Step 3:

[0633] The server uses an emotion engine to recognize the user's emotions from the extracted facial expression data. It identifies the type of emotion (e.g., joy, sadness, surprise, etc.) and analyzes this information in conjunction with the characteristics of sign language.

[0634] Step 4:

[0635] The server generates corresponding natural language text based on the characteristics and emotional information of the sign language. The emotional information is reflected in the text to ensure that the expression includes emotional nuances.

[0636] Step 5:

[0637] The server converts the generated text data into synthesized speech. Based on emotional information, it adjusts the tone and speed of the speech to create speech data that conveys emotion.

[0638] Step 6:

[0639] The server sends synthesized speech data to the terminal, and the terminal plays the received speech to the user through its speaker. A healthy person receives the emotionally charged sign language content as speech.

[0640] Step 7:

[0641] The device captures speech uttered by a healthy individual using its microphone and sends that audio data to a server.

[0642] Step 8:

[0643] The server uses speech recognition technology to convert the audio data into text, analyzes its content, and estimates the emotions conveyed.

[0644] Step 9:

[0645] The server generates sign language action data with appropriate emotional expressions based on estimated emotional information. It then creates sign language animations that reflect emotions by referring to a sign language database.

[0646] Step 10:

[0647] The server sends the generated sign language animation data to the terminal.

[0648] Step 11:

[0649] The device displays sign language animations on its screen, conveying emotions through sign language in a way that is visible to the hearing impaired. As a result, users can visually understand emotions along with the audio content.

[0650] (Example 2)

[0651] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0652] Conventional sign language translation systems failed to consider the user's emotions when converting sign language movements into natural language, resulting in a lack of emotional transmission in communication. Furthermore, even when converting spoken language from hearing individuals into sign language, generating sign language movements that reflected appropriate emotions was difficult. This made it challenging to accurately convey intentions, especially in conversations where emotional nuances are crucial.

[0653] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0654] In this invention, the server includes means for acquiring video data to recognize the user's emotions in addition to sign language; means for extracting the characteristics of the sign language and emotions based on the user's facial expressions from the acquired video data; and means for analyzing the extracted characteristics of the sign language and emotions and converting them into the corresponding natural language while taking emotions into consideration. This enables rich, two-way communication, including emotions, between sign language and natural language.

[0655] Sign language is a visual language that involves hand and arm movements and facial expressions, used by people who are deaf or hard of hearing to communicate.

[0656] A "user" refers to a person who uses this system to communicate, engaging in dialogue through sign language or voice.

[0657] "Video data" refers to dynamic data that visually records sign language and facial expressions, acquired using imaging devices such as cameras.

[0658] "Emotions" refer to the psychological state that can be gleaned from a user's facial expressions and voice, and include information that encompasses nuances such as joy, anger, and sadness.

[0659] "Natural language" is a general term for spoken and written languages ​​that humans use on a daily basis, and it is a language that conveys meaning based on grammar.

[0660] "Audio output" refers to the process of playing back natural language generated by the server as audio, making it available for listening.

[0661] "Animation" is a type of video that visually represents movement by sequentially playing a series of still images, and is used to represent the movements of sign language.

[0662] This invention improves the quality of communication by incorporating an emotion recognition function into a system for mutual conversion between sign language and natural language. This system consists of a terminal that simultaneously captures the user's sign language and facial expressions, and a server that analyzes and converts that data.

[0663] The device has a built-in camera that captures the user's sign language movements and facial expressions in real time. This video data is compressed and sent to a server over the network. The device includes a high-resolution camera for proper video data capture and a high-performance chip for smooth processing.

[0664] The server uses advanced video processing software and machine learning algorithms to analyze the received video data. Specifically, it analyzes sign language movements and facial expressions to extract the content and emotions of the sign language. The server also uses an emotion engine to identify emotions such as smiles, anger, and sadness. This emotion information is taken into consideration during the natural language conversion process. Furthermore, a generative AI model is used to generate natural language text based on the extracted sign language and emotion information. This text is then converted into speech by a speech synthesis engine via a natural language processing (NLP) engine and sent to the terminal.

[0665] For example, if a user smiles while showing gratitude in sign language, the server will convert this into audio expressing the joyful emotion of "thank you." The terminal will then play this audio and relay it to a hearing person. An example of a prompt in this process would be: "Video has been captured of a user smiling and saying 'thank you' in sign language. Please translate this sign language appropriately and generate natural-sounding audio that includes the emotion of a smile."

[0666] The system also acquires voice input from sighted individuals and converts it to text through speech recognition on the server. It then generates and displays appropriate emotionally charged sign language animations to the user. This enables emotionally engaging two-way communication between visually impaired and sighted individuals.

[0667] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0668] Step 1:

[0669] The user performs a sign language and facial expression routine. Subsequently, the device's camera simultaneously captures the user's sign language movements and facial expressions. Video data is the input, and data containing hand and facial movement information is output. This data is appropriately compressed by a codec and sent to the server.

[0670] Step 2:

[0671] The server decodes the received compressed video data and uses machine learning algorithms to extract features from sign language movements and facial expressions. Here, video analysis software is used to extract meaningful hand and facial movements from the video data. This becomes the input, and sign language movement features and facial expression data are output.

[0672] Step 3:

[0673] The server inputs the extracted features into the emotion engine to recognize the user's emotional state. The emotion engine identifies emotions such as joy and anger based on changes in facial expressions, particularly eyebrow and mouth movements. The input consists of sign language gesture features and facial expression data, and the output is emotional information.

[0674] Step 4:

[0675] The server fuses the analyzed sign language features and emotional information, and uses a generative AI model to convert this into natural language text. This process generates appropriate sentences that retain the meaning and emotional nuances of the sign language. The input is sign language features and emotional information, and the output is natural language text.

[0676] Step 5:

[0677] The server inputs the generated natural language text into a speech synthesis engine and converts it into a human-readable speech format. Speech synthesis generates speech that reflects emotions based on the content of the text. The input is natural language text, and the output is synthesized speech data.

[0678] Step 6:

[0679] The device receives synthesized speech transmitted from the server and plays it back through its speaker. Here, audio data serves as input, and an emotionally charged audio output is produced for a hearing person. The device effectively conveys the content and emotions of sign language through audio playback.

[0680] Step 7:

[0681] The terminal captures voice input from a healthy individual via a microphone and sends that data to a server. The audio data becomes the input, and the audio data is output in a format that is then sent to the server.

[0682] Step 8:

[0683] The server analyzes the audio sent to it using a speech recognition engine and converts it into natural language text. The input is audio data, and the output is the converted text.

[0684] Step 9:

[0685] The server generates sign language animations that include appropriate emotional nuances based on natural language text. A generative AI model is used to create motion data from text. The input is natural language text, and the output is sign language animation data.

[0686] Step 10:

[0687] The terminal plays sign language animation data received from the server on the screen. This allows the user to visually understand messages from hearing individuals. The input is sign language animation data, and the output is the movements displayed on the screen.

[0688] (Application Example 2)

[0689] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0690] In modern times, accurately conveying emotional nuances and intentions is difficult in communication between sign language users and those who primarily use spoken language. This is especially true in caregiving settings, where understanding the emotions of the service user is crucial, and appropriate responses based on that understanding are required. In this context, there is a need to provide a bidirectional conversion system between natural language and sign language that takes emotions into consideration.

[0691] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0692] In this invention, the server includes means for acquiring information, means for extracting sign language features from the acquired information, means for analyzing the extracted features and recognizing emotions, means for converting the emotions into the corresponding natural language, and means for outputting the converted natural language data as speech. This enables two-way communication between sign language and natural language that takes emotions into account.

[0693] "Means of acquiring information" refers to devices and technologies for capturing data related to communication, such as sign language and voice.

[0694] "Methods for extracting the characteristics of sign language" refer to technologies that identify the characteristics of hand movements and facial expressions from acquired video data and extract them in an analyzable format.

[0695] "Means of recognizing emotions" refers to technologies that analyze and identify a user's emotions from sign language or voice data.

[0696] "Methods for converting into natural language while reflecting emotions" refers to technologies that convert sign language or speech into corresponding natural language text while taking emotions into consideration.

[0697] "Means for outputting natural language data as speech" refers to technology that converts text data into speech and provides it to the user in an audible format.

[0698] "Means for acquiring speech and converting it into natural language that takes emotions into account" refers to technology that recognizes speech data and converts it into natural language text that reflects the speaker's emotions.

[0699] "A means of outputting sign language motion data as animation" refers to a technology that visually represents data converted into sign language and presents it to the user as animation.

[0700] "Means for learning individual user emotional patterns" refers to a technology that accumulates each user's unique emotional patterns to improve the system's conversion accuracy.

[0701] The system for implementing this invention is designed to enable emotionally rich communication between sign language users and spoken language users in nursing care facilities. Specific embodiments of the system are described below.

[0702] System configuration:

[0703] The server has high processing power to receive and analyze sign language and speech data, and uses open-source image processing libraries (e.g., OpenCV, dlib) and emotion recognition APIs (e.g., Microsoft Azure Emotion API). It also utilizes AI services (e.g., Google Cloud Text-to-Speech API) for natural language processing and speech synthesis.

[0704] The terminals are portable devices such as smartphones and tablets that are equipped with cameras and microphones. This allows the terminals to directly acquire sign language and voice data and transmit it to the server in real time.

[0705] The users include both individuals who primarily use sign language and staff who use spoken language, and they communicate interactively through the system.

[0706] Example of operation:

[0707] When a user conveys "thank you" in sign language, the data, including their facial expressions, is sent from the device to the server. The server recognizes the emotion from the sign language data and facial expressions and converts it into natural language. Then, the message "thank you" is played back from the device's speaker in a cheerful voice that reflects the emotion. Conversely, if the message "that's wonderful" is conveyed verbally, it is presented to the user as an emotionally charged sign language animation.

[0708] Example prompts for generative AI models:

[0709] "If a user makes a smiling face and signs 'thank you,' what tone of voice will be generated?"

[0710] This invention aims to enrich communication in caregiving settings by enabling step-by-step information exchange between sign language and natural language.

[0711] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0712] Step 1:

[0713] The device uses its built-in camera to capture the user's sign language and facial expressions in real time. The frames captured as video data become the input. This data is then sent to the server for the next step.

[0714] Step 2:

[0715] The server receives the transmitted video data and uses an image processing library (e.g., OpenCV) to extract the characteristics of sign language from hand movements and facial features. During this process, specific motion data and facial expression data of the sign language are output.

[0716] Step 3:

[0717] The server uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to analyze the user's emotions from the extracted facial expression data. This process outputs information about the user's emotional state, which is then used in subsequent processing.

[0718] Step 4:

[0719] The server converts sign language motion data and user emotion information into natural language text. To reflect emotions, a generative AI model is used to adjust the nuances of the natural language. The resulting natural language text is then output.

[0720] Step 5:

[0721] The server converts natural language text into speech using the Google Cloud Text-to-Speech API. The generated speech data is output and sent to the device.

[0722] Step 6:

[0723] The terminal plays the audio data received from the server through its speaker. In this step, the user is provided with audio information that reflects emotions.

[0724] Step 7:

[0725] Conversely, the device uses its microphone to capture the voice of a healthy person and sends the audio data to the server. In this case, the input is audio data.

[0726] Step 8:

[0727] The server uses speech recognition technology to convert speech data into text data and then performs emotion recognition. The analysis results are output as natural language text and emotion information.

[0728] Step 9:

[0729] The server generates appropriate emotional sign language action data based on natural language text and emotional information. This results in the output of sign language animation data, which is then sent to the terminal.

[0730] Step 10:

[0731] The terminal plays the received sign language animation data on the screen, providing it in a visible format to the user communicating in sign language. This completes the entire process.

[0732] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0733] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0734] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0735] [Fourth Embodiment]

[0736] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0737] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0738] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0739] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0740] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0741] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0742] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0743] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0744] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0745] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0746] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0747] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0748] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0749] The real-time sign language translation system according to the present invention is a system that enables smooth communication between hearing individuals who do not understand sign language and hearing-impaired individuals. This system acquires sign language video in real time and converts it into speech or text to convey the intended message. It also has the function of converting the speech of hearing individuals into sign language and conveying it visually.

[0750] The device captures the user's sign language in real time via its camera. The video data is transmitted to the server with low latency. The server analyzes the received data and extracts the characteristics of the sign language. This identifies which sign language was shown and converts it into corresponding natural language text. The converted text is output as speech using speech synthesis technology and played back on the device.

[0751] Conversely, the device acquires the voice spoken by a hearing person using a microphone and sends that voice data to a server. The server converts the voice data into text using speech recognition technology, and then converts the text into appropriate sign language movements. These sign language movements are displayed as animations on the device, conveying the intended message to the hearing impaired.

[0752] As a concrete example, if a user (who is hearing impaired) expresses "Good morning" in sign language, the terminal sends the video to the server, which analyzes the sign language and converts it into the text "Good morning." This text is then output as audio and instantly transmitted to a hearing person.

[0753] This system goes beyond simple sign language-to-speech conversion; it has the ability to learn each user's individual sign language style and speech patterns, improving its conversion accuracy over time. This enables highly accurate translations, making it useful in a variety of communication situations.

[0754] The following describes the processing flow.

[0755] Step 1:

[0756] The device captures the user's sign language in real time via its camera. It acquires the video at a high frame rate and stores the visual information as digital data.

[0757] Step 2:

[0758] The terminal compresses and encodes the captured video data and transmits it to the server via the communication line with low latency.

[0759] Step 3:

[0760] The server extracts features such as hand position, movement, and finger shape from the received video data. It uses computer vision algorithms to identify the necessary feature points.

[0761] Step 4:

[0762] The server uses a deep learning model to recognize sign language based on the extracted feature points. Using the trained model, it generates corresponding natural language text data.

[0763] Step 5:

[0764] The server uses natural language processing technology to generate synthesized speech from the obtained text data. This creates audio output as audio data.

[0765] Step 6:

[0766] The server sends the generated audio data to the terminal.

[0767] Step 7:

[0768] The device plays the received audio data through its speaker, conveying the content of the sign language to a hearing person as audio.

[0769] Step 8:

[0770] The device acquires the voice spoken by a healthy person through its microphone and sends the voice data to the server.

[0771] Step 9:

[0772] The server converts the received audio data into text using speech recognition technology. By obtaining the text data, it prepares for the next processing step.

[0773] Step 10:

[0774] The server converts the obtained text data into sign language motion data. It then refers to a sign language database to generate specific sign language animations.

[0775] Step 11:

[0776] The server sends the generated sign language animation data to the terminal.

[0777] Step 12:

[0778] The device displays received sign language animations on its screen, visually conveying messages to people with hearing impairments.

[0779] (Example 1)

[0780] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0781] Facilitating smooth communication between deaf individuals who use sign language and hearing individuals who do not understand sign language is difficult. Conventional systems struggle to provide accurate real-time translation, thus creating a need for technology that efficiently performs bidirectional conversion between sign language and speech or text.

[0782] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0783] In this invention, the server includes a device for acquiring video information, means for extracting sign language features from the acquired video information, and a mechanism for analyzing the extracted features and converting them into corresponding natural language. This makes it possible to convert sign language into natural language in real time with high accuracy and output it as speech.

[0784] "Visual information" refers to dynamic visual data acquired to represent sign language movements.

[0785] "Characteristics of sign language" refer to identifiable elements such as the shape, movement, and position of the hands in sign language actions.

[0786] "Natural language" refers to the language that humans use for normal communication, and which can be expressed as spoken or written language.

[0787] "Synthesized speech" is a technology that converts natural language text information into audio signals and plays them back using mechanically generated speech.

[0788] "Audio information" refers to data obtained from speech and is used for speech recognition processing.

[0789] "Sign language motion information" refers to the motion data that makes up sign language animations generated based on audio and text.

[0790] "Animation" refers to a series of visual representations used to visually represent sign language and its movements.

[0791] "Individual user characteristics" refers to the unique actions and vocal patterns of each user in sign language and speech.

[0792] A "learning device" is a processing device that continuously learns the characteristics of the user's sign language and speech to improve conversion accuracy.

[0793] The real-time sign language translation system according to this invention aims to enable smooth communication between hearing-impaired individuals who use sign language and hearing individuals who do not understand sign language. Specific embodiments for carrying out this invention are described below.

[0794] The device uses a camera to capture the user's sign language movements in real time. The video information is processed by specialized equipment such as a Movidius Neural Compute Stick and transmitted to a server with low latency. The video information acquired here is high-quality visual data for accurately capturing sign language movements.

[0795] The server analyzes the received video information using a deep learning model. A CNN model utilizing TensorFlow is effective as the model, and this model functions as a mechanism to extract sign language features and convert them into corresponding natural language. In this process, the server identifies sign language features such as hand shape, movement, and position. The natural language converted based on the analysis is output as synthesized speech using speech synthesis technology such as the Google Text-to-Speech API.

[0796] Conversely, voice information spoken by a healthy person through the microphone on the device is also sent to the server. The server uses the Google Speech-to-Text API to convert this voice information into natural language text. This is then converted into sign language gesture information and visually displayed on the device using animation software such as Unity3D.

[0797] As a concrete example, if a user (who is hearing impaired) expresses "thank you" in sign language, the device sends the video to the server. The server analyzes the sign language, converts it into natural language text "thank you," and outputs it as synthesized speech to a hearing person.

[0798] One possible prompt to input into the generative AI model is: "Capture a video using a specific sign language action as an example, and explain how it is converted into speech."

[0799] This system also has a function that improves conversion accuracy by learning the user's individual sign language style and voice patterns over time, and is expected to be used in an even wider range of communication scenarios.

[0800] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0801] Step 1:

[0802] The device captures the user's sign language movements in real time via its camera. During this process, the device acquires video information and records the images at a high frame rate. The input is the user's sign language movements, and the output is raw video data.

[0803] Step 2:

[0804] The terminal compresses the acquired video information into the required format and sends it to the server. Here, the video is optimized to streamline data transfer and enable low-latency communication. The input is raw video data, and the output is compressed video data.

[0805] Step 3:

[0806] The server receives the transmitted compressed video data and extracts sign language features using a deep learning model. Specifically, a model using TensorFlow analyzes the shape and movement of the hands and recognizes feature points based on that analysis. The input is compressed video data, and the output is sign language feature information.

[0807] Step 4:

[0808] The server utilizes a sign language-to-text mapping database to convert extracted sign language feature information into natural language. This analysis converts the intent conveyed by the sign language into natural language sentences. The input is sign language feature information, and the output is natural language text data.

[0809] Step 5:

[0810] The server converts the translated natural language text into speech data using speech synthesis technology. The Google Text-to-Speech API is used for the speech generation process. The input is natural language text data, and the output is synthesized speech.

[0811] Step 6:

[0812] The device receives synthesized speech transmitted from the server and plays the speech through its built-in speaker. This allows hearing individuals to understand the meaning of sign language as spoken language. The input is synthesized speech data, and the output is spoken audio.

[0813] Step 7:

[0814] The device captures voice information spoken by a healthy individual using a microphone. The acquired voice information is recorded in clear quality. The input is the voice of a healthy individual, and the output is voice data.

[0815] Step 8:

[0816] The terminal optimizes and compresses the recorded audio data before sending it to the server. It then transmits it with low latency. The input is audio data, and the output is compressed audio data.

[0817] Step 9:

[0818] The server converts the received audio data into natural language text using speech recognition technology. The Google Speech-to-Text API analyzes the content of the audio and converts it into text format. The input is compressed audio data, and the output is text data.

[0819] Step 10:

[0820] The server converts text data into sign language motion information. In this process, it uses an animation engine to convert the text into motion data in order to generate corresponding sign language animations. The input is text data, and the output is sign language motion information.

[0821] Step 11:

[0822] The terminal displays sign language gesture information received from the server as an animation on its screen. This allows the user (a person with hearing impairment) to receive messages visually. The input is sign language gesture information, and the output is a sign language animation.

[0823] (Application Example 1)

[0824] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0825] In modern society, the lack of effective means of smooth communication between the hearing impaired and hearing individuals remains a significant challenge. For the hearing impaired, whose primary means of communication is sign language, communication with hearing individuals is often difficult in many situations in daily life. Therefore, there is a need for real-time conversion between sign language and spoken language to achieve smooth and accurate communication. Furthermore, providing conversion accuracy optimized for each individual user is also crucial.

[0826] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0827] In this invention, the server includes means for acquiring video data for recognizing sign language, means for acquiring audio data and converting it into natural language, and means for learning user characteristics using feedback information and improving conversion accuracy. This enables real-time and highly accurate communication between hearing-impaired and hearing individuals.

[0828] "Means for acquiring video data for recognizing sign language" refers to methods for recording sign language movements in real time using a camera mounted on a terminal or device.

[0829] "Methods for extracting the characteristics of sign language" refer to algorithms that identify specific hand shapes and movement patterns from acquired sign language video data and analyze the meaning of the sign language.

[0830] "Means for converting to corresponding natural language" refers to methods for converting extracted sign language features into words and treating them as text data with concrete meaning.

[0831] "Means for converting converted natural language data into speech and outputting it via a speech output device" refers to a technology for converting generated text data into sound waves using speech synthesis technology and then playing them back through an acoustic device.

[0832] "Means for acquiring audio data and converting it into natural language" refers to speech recognition technology that analyzes input audio data and expresses its content as text.

[0833] "Means for converting natural language into sign language motion data" refers to algorithms for converting natural language text information into sequences of actions that can be communicated in sign language.

[0834] "Means for displaying sign language motion data on a visual display device" refers to a display technology that animates generated sign language motions and presents them visually.

[0835] "Methods for learning user characteristics using feedback information and improving conversion accuracy" refers to a function that analyzes the user's usage history and feedback data, optimizes the conversion algorithm, and achieves highly accurate translations tailored to individual users.

[0836] This invention is a system for facilitating communication using sign language. The system mainly consists of a terminal worn by the user and a server connected to it.

[0837] The devices are smart glasses or other mobile devices. They have built-in cameras and microphones that capture the user's sign language and voice, respectively. The sign language video data captured by the camera is transmitted to a server with low latency. The server analyzes the acquired video using machine learning models such as TensorFlow Lite to extract sign language features. This analysis identifies the meaning of the sign language movements. The identified sign language features are converted into natural language text and then spoken using speech synthesis technology. This allows the information to be conveyed to hearing individuals in both text and audio formats.

[0838] Conversely, voice input from able-bodied individuals is captured by the device and sent to the server. The server uses the Google Cloud Speech-to-Text API to convert this voice data into natural language text. Then, a generative AI model is used to convert the text into sign language motion data. This sign language motion is then animated using software such as Unity and displayed on the device's screen.

[0839] Furthermore, the system learns user characteristics and improves translation accuracy by receiving feedback as needed. This enables the achievement of highly accurate translations optimized for individual users.

[0840] For example, if a user (who is hearing impaired) uses sign language to mean "turn up the TV volume," the server analyzes this sign language and converts it into the natural language phrase "Please turn up the TV volume." Conversely, if a hearing person says "Good morning," the system animates this speech as sign language and conveys it to the user. An example of a prompt message to the generative AI model would be: "Analyze the sign language actions captured by the user via the camera and output their intent as spoken language. Also, display the spoken language as a sign language animation on the screen."

[0841] This system makes it possible for both people with hearing impairments and those without hearing impairments to communicate without difficulty.

[0842] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0843] Step 1:

[0844] The device captures the user's sign language expressions with its camera and temporarily stores the video data. Next, it transmits this video data to the server with low latency. The input is the user's sign language actions, and the output is the transfer of video data to the server.

[0845] Step 2:

[0846] The server uses TensorFlow Lite to extract sign language features from the received video data. The input is captured video data of sign language, and the output is a sign language feature vector. This vector is used to analyze the content of the sign language and convert it into natural language text.

[0847] Step 3:

[0848] The server converts the generated natural language text into speech using speech synthesis technology (e.g., a text-to-speech engine). The input is natural language text, and the output is audio data. This audio data is sent to the terminal and played back via an audio output device.

[0849] Step 4:

[0850] In the reverse process, the terminal acquires voice from a healthy person using a microphone and sends that voice data to the server. The input is the voice of the healthy person, and the output is the transfer of voice data to the server.

[0851] Step 5:

[0852] The server uses the Google Cloud Speech-to-Text API to convert speech data into natural language text. The input is speech data from a healthy individual, and the output is natural language text. Based on this text, a generative AI model is used to convert the text into sign language gesture data.

[0853] Step 6:

[0854] The server animates the converted sign language motion data using software such as Unity, and sends this animated data to the terminal. The input is the sign language motion data, and the output is the transmission of the animated sign language data to the terminal.

[0855] Step 7:

[0856] The terminal displays animated sign language to the user. The input is animation data from the server, and the output is sign language information visually communicated to the user.

[0857] Step 8:

[0858] Based on user and healthy user feedback, the server updates the conversion model and learns to achieve optimized conversion accuracy for each user. The input is the feedback data and existing model parameters, and the output is the optimized conversion model.

[0859] This process enables real-time translation between sign language and spoken language, thereby lowering communication barriers.

[0860] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0861] This invention aims to achieve richer communication by adding a function to recognize user emotions to a system that performs mutual conversion between sign language and natural language. This system is equipped with an emotion engine that recognizes emotions from the user's facial expressions and voice during the process of extracting the characteristics of sign language, and can convert them into natural language text while taking those emotions into consideration.

[0862] The device captures the user's sign language and facial expressions using its camera. The video data is sent to a server, where characteristic hand and facial movements are analyzed. In addition to the content of the sign language, the server uses an emotion engine to recognize the user's emotions, identifying expressions such as smiles, anger, and sadness. This extracts the nuances of emotion conveyed in the spoken content.

[0863] Based on the analyzed data, the server generates natural language text that reflects emotions and outputs it as synthesized speech. The terminal plays the received audio through its speaker, conveying the content of the sign language and the emotions behind it as audio information to a hearing person. For example, if a user signs "hello" with a smile, the audio output will say "hello" in a cheerful tone.

[0864] Furthermore, when the device acquires the voice of a healthy person, the server converts the voice into text and generates appropriate sign language actions with corresponding emotions from that text. The generated sign language data is sent to the device as an animation and displayed to the user. At this time, based on the emotion data, for example, a word pronounced with emotion, such as "wonderful," will be reproduced with a sign language animation that conveys the appropriate emotion.

[0865] By incorporating emotion recognition in this way, it becomes possible to enable more emotionally rich two-way communication between natural language and sign language than before. This feature will be particularly useful for users in situations where complex emotions need to be conveyed, or in informal communication.

[0866] The following describes the processing flow.

[0867] Step 1:

[0868] The device uses a camera to simultaneously capture the user's sign language movements and facial expressions. High-resolution video data is recorded in real time and transmitted to a server via the network.

[0869] Step 2:

[0870] The server extracts features corresponding to sign language movements from the received video data. In addition to hand position and movement, it analyzes facial expression data to capture emotional elements along with the sign language.

[0871] Step 3:

[0872] The server uses an emotion engine to recognize the user's emotions from the extracted facial expression data. It identifies the type of emotion (e.g., joy, sadness, surprise, etc.) and analyzes this information in conjunction with the characteristics of sign language.

[0873] Step 4:

[0874] The server generates corresponding natural language text based on the characteristics and emotional information of the sign language. The emotional information is reflected in the text to ensure that the expression includes emotional nuances.

[0875] Step 5:

[0876] The server converts the generated text data into synthesized speech. Based on emotional information, it adjusts the tone and speed of the speech to create speech data that conveys emotion.

[0877] Step 6:

[0878] The server sends synthesized speech data to the terminal, and the terminal plays the received speech to the user through its speaker. A healthy person receives the emotionally charged sign language content as speech.

[0879] Step 7:

[0880] The device captures speech uttered by a healthy individual using its microphone and sends that audio data to a server.

[0881] Step 8:

[0882] The server uses speech recognition technology to convert the audio data into text, analyzes its content, and estimates the emotions conveyed.

[0883] Step 9:

[0884] The server generates sign language action data with appropriate emotional expressions based on estimated emotional information. It then creates sign language animations that reflect emotions by referring to a sign language database.

[0885] Step 10:

[0886] The server sends the generated sign language animation data to the terminal.

[0887] Step 11:

[0888] The device displays sign language animations on its screen, conveying emotions through sign language in a way that is visible to the hearing impaired. As a result, users can visually understand emotions along with the audio content.

[0889] (Example 2)

[0890] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0891] Conventional sign language translation systems failed to consider the user's emotions when converting sign language movements into natural language, resulting in a lack of emotional transmission in communication. Furthermore, even when converting spoken language from hearing individuals into sign language, generating sign language movements that reflected appropriate emotions was difficult. This made it challenging to accurately convey intentions, especially in conversations where emotional nuances are crucial.

[0892] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0893] In this invention, the server includes means for acquiring video data to recognize the user's emotions in addition to sign language; means for extracting the characteristics of the sign language and emotions based on the user's facial expressions from the acquired video data; and means for analyzing the extracted characteristics of the sign language and emotions and converting them into the corresponding natural language while taking emotions into consideration. This enables rich, two-way communication, including emotions, between sign language and natural language.

[0894] Sign language is a visual language that involves hand and arm movements and facial expressions, used by people who are deaf or hard of hearing to communicate.

[0895] A "user" refers to a person who uses this system to communicate, engaging in dialogue through sign language or voice.

[0896] "Video data" refers to dynamic data that visually records sign language and facial expressions, acquired using imaging devices such as cameras.

[0897] "Emotions" refer to the psychological state that can be gleaned from a user's facial expressions and voice, and include information that encompasses nuances such as joy, anger, and sadness.

[0898] "Natural language" is a general term for spoken and written languages ​​that humans use on a daily basis, and it is a language that conveys meaning based on grammar.

[0899] "Audio output" refers to the process of playing back natural language generated by the server as audio, making it available for listening.

[0900] "Animation" is a type of video that visually represents movement by sequentially playing a series of still images, and is used to represent the movements of sign language.

[0901] This invention improves the quality of communication by incorporating an emotion recognition function into a system for mutual conversion between sign language and natural language. This system consists of a terminal that simultaneously captures the user's sign language and facial expressions, and a server that analyzes and converts that data.

[0902] The device has a built-in camera that captures the user's sign language movements and facial expressions in real time. This video data is compressed and sent to a server over the network. The device includes a high-resolution camera for proper video data capture and a high-performance chip for smooth processing.

[0903] The server uses advanced video processing software and machine learning algorithms to analyze the received video data. Specifically, it analyzes sign language movements and facial expressions to extract the content and emotions of the sign language. The server also uses an emotion engine to identify emotions such as smiles, anger, and sadness. This emotion information is taken into consideration during the natural language conversion process. Furthermore, a generative AI model is used to generate natural language text based on the extracted sign language and emotion information. This text is then converted into speech by a speech synthesis engine via a natural language processing (NLP) engine and sent to the terminal.

[0904] For example, if a user smiles while showing gratitude in sign language, the server will convert this into audio expressing the joyful emotion of "thank you." The terminal will then play this audio and relay it to a hearing person. An example of a prompt in this process would be: "Video has been captured of a user smiling and saying 'thank you' in sign language. Please translate this sign language appropriately and generate natural-sounding audio that includes the emotion of a smile."

[0905] The system also acquires voice input from sighted individuals and converts it to text through speech recognition on the server. It then generates and displays appropriate emotionally charged sign language animations to the user. This enables emotionally engaging two-way communication between visually impaired and sighted individuals.

[0906] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0907] Step 1:

[0908] The user performs a sign language and facial expression routine. Subsequently, the device's camera simultaneously captures the user's sign language movements and facial expressions. Video data is the input, and data containing hand and facial movement information is output. This data is appropriately compressed by a codec and sent to the server.

[0909] Step 2:

[0910] The server decodes the received compressed video data and uses machine learning algorithms to extract features from sign language movements and facial expressions. Here, video analysis software is used to extract meaningful hand and facial movements from the video data. This becomes the input, and sign language movement features and facial expression data are output.

[0911] Step 3:

[0912] The server inputs the extracted features into the emotion engine to recognize the user's emotional state. The emotion engine identifies emotions such as joy and anger based on changes in facial expressions, particularly eyebrow and mouth movements. The input consists of sign language gesture features and facial expression data, and the output is emotional information.

[0913] Step 4:

[0914] The server fuses the analyzed sign language features and emotional information, and uses a generative AI model to convert this into natural language text. This process generates appropriate sentences that retain the meaning and emotional nuances of the sign language. The input is sign language features and emotional information, and the output is natural language text.

[0915] Step 5:

[0916] The server inputs the generated natural language text into a speech synthesis engine and converts it into a human-readable speech format. Speech synthesis generates speech that reflects emotions based on the content of the text. The input is natural language text, and the output is synthesized speech data.

[0917] Step 6:

[0918] The device receives synthesized speech transmitted from the server and plays it back through its speaker. Here, audio data serves as input, and an emotionally charged audio output is produced for a hearing person. The device effectively conveys the content and emotions of sign language through audio playback.

[0919] Step 7:

[0920] The terminal captures voice input from a healthy individual via a microphone and sends that data to a server. The audio data becomes the input, and the audio data is output in a format that is then sent to the server.

[0921] Step 8:

[0922] The server analyzes the audio sent to it using a speech recognition engine and converts it into natural language text. The input is audio data, and the output is the converted text.

[0923] Step 9:

[0924] The server generates sign language animations that include appropriate emotional nuances based on natural language text. A generative AI model is used to create motion data from text. The input is natural language text, and the output is sign language animation data.

[0925] Step 10:

[0926] The terminal plays sign language animation data received from the server on the screen. This allows the user to visually understand messages from hearing individuals. The input is sign language animation data, and the output is the movements displayed on the screen.

[0927] (Application Example 2)

[0928] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0929] In modern times, accurately conveying emotional nuances and intentions is difficult in communication between sign language users and those who primarily use spoken language. This is especially true in caregiving settings, where understanding the emotions of the service user is crucial, and appropriate responses based on that understanding are required. In this context, there is a need to provide a bidirectional conversion system between natural language and sign language that takes emotions into consideration.

[0930] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0931] In this invention, the server includes means for acquiring information, means for extracting sign language features from the acquired information, means for analyzing the extracted features and recognizing emotions, means for converting the emotions into the corresponding natural language, and means for outputting the converted natural language data as speech. This enables two-way communication between sign language and natural language that takes emotions into account.

[0932] "Means of acquiring information" refers to devices and technologies for capturing data related to communication, such as sign language and voice.

[0933] "Methods for extracting the characteristics of sign language" refer to technologies that identify the characteristics of hand movements and facial expressions from acquired video data and extract them in an analyzable format.

[0934] "Means of recognizing emotions" refers to technologies that analyze and identify a user's emotions from sign language or voice data.

[0935] "Methods for converting into natural language while reflecting emotions" refers to technologies that convert sign language or speech into corresponding natural language text while taking emotions into consideration.

[0936] "Means for outputting natural language data as speech" refers to technology that converts text data into speech and provides it to the user in an audible format.

[0937] "Means for acquiring speech and converting it into natural language that takes emotions into account" refers to technology that recognizes speech data and converts it into natural language text that reflects the speaker's emotions.

[0938] "A means of outputting sign language motion data as animation" refers to a technology that visually represents data converted into sign language and presents it to the user as animation.

[0939] "Means for learning individual user emotional patterns" refers to a technology that accumulates each user's unique emotional patterns to improve the system's conversion accuracy.

[0940] The system for implementing this invention is designed to enable emotionally rich communication between sign language users and spoken language users in nursing care facilities. Specific embodiments of the system are described below.

[0941] System configuration:

[0942] The server has high processing power to receive and analyze sign language and speech data, and uses open-source image processing libraries (e.g., OpenCV, dlib) and emotion recognition APIs (e.g., Microsoft Azure Emotion API). It also utilizes AI services (e.g., Google Cloud Text-to-Speech API) for natural language processing and speech synthesis.

[0943] The terminals are portable devices such as smartphones and tablets that are equipped with cameras and microphones. This allows the terminals to directly acquire sign language and voice data and transmit it to the server in real time.

[0944] The users include both individuals who primarily use sign language and staff who use spoken language, and they communicate interactively through the system.

[0945] Example of operation:

[0946] When a user conveys "thank you" in sign language, the data, including their facial expressions, is sent from the device to the server. The server recognizes the emotion from the sign language data and facial expressions and converts it into natural language. Then, the message "thank you" is played back from the device's speaker in a cheerful voice that reflects the emotion. Conversely, if the message "that's wonderful" is conveyed verbally, it is presented to the user as an emotionally charged sign language animation.

[0947] Example prompts for generative AI models:

[0948] "If a user makes a smiling face and signs 'thank you,' what tone of voice will be generated?"

[0949] This invention aims to enrich communication in caregiving settings by enabling step-by-step information exchange between sign language and natural language.

[0950] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0951] Step 1:

[0952] The device uses its built-in camera to capture the user's sign language and facial expressions in real time. The frames captured as video data become the input. This data is then sent to the server for the next step.

[0953] Step 2:

[0954] The server receives the transmitted video data and uses an image processing library (e.g., OpenCV) to extract the characteristics of sign language from hand movements and facial features. During this process, specific motion data and facial expression data of the sign language are output.

[0955] Step 3:

[0956] The server uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to analyze the user's emotions from the extracted facial expression data. This process outputs information about the user's emotional state, which is then used in subsequent processing.

[0957] Step 4:

[0958] The server converts sign language motion data and user emotion information into natural language text. To reflect emotions, a generative AI model is used to adjust the nuances of the natural language. The resulting natural language text is then output.

[0959] Step 5:

[0960] The server converts natural language text into speech using the Google Cloud Text-to-Speech API. The generated speech data is output and sent to the device.

[0961] Step 6:

[0962] The terminal plays the audio data received from the server through its speaker. In this step, the user is provided with audio information that reflects emotions.

[0963] Step 7:

[0964] Conversely, the device uses its microphone to capture the voice of a healthy person and sends the audio data to the server. In this case, the input is audio data.

[0965] Step 8:

[0966] The server uses speech recognition technology to convert speech data into text data and then performs emotion recognition. The analysis results are output as natural language text and emotion information.

[0967] Step 9:

[0968] The server generates appropriate emotional sign language action data based on natural language text and emotional information. This results in the output of sign language animation data, which is then sent to the terminal.

[0969] Step 10:

[0970] The terminal plays the received sign language animation data on the screen, providing it in a visible format to the user communicating in sign language. This completes the entire process.

[0971] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0972] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0973] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0974] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0975] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0976] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0977] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0978] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0979] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0980] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0981] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0982] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0983] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0984] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0985] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0986] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0987] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0988] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0989] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0990] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0991] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0992] The following is further disclosed regarding the embodiments described above.

[0993] (Claim 1)

[0994] A means of acquiring video data for recognizing sign language,

[0995] A method for extracting the characteristics of sign language from acquired video data,

[0996] A means of analyzing extracted features and converting them into corresponding natural language,

[0997] A means of outputting the converted natural language data,

[0998] A system that includes this.

[0999] (Claim 2)

[1000] A means of acquiring audio data and converting it into natural language,

[1001] A means of converting the converted natural language into sign language motion data,

[1002] A means for outputting converted sign language motion data,

[1003] The system according to claim 1, including the following:

[1004] (Claim 3)

[1005] The system according to claim 1, further comprising a learning means for learning the individual characteristics of users and improving conversion accuracy.

[1006] "Example 1"

[1007] (Claim 1)

[1008] A device for acquiring video information,

[1009] A method for extracting the characteristics of sign language from acquired video information,

[1010] A mechanism that analyzes extracted features and converts them into corresponding natural language,

[1011] A device that outputs converted natural language information as synthesized speech,

[1012] An information processing system that includes this.

[1013] (Claim 2)

[1014] A means of acquiring audio information and converting it into natural language,

[1015] A device that converts converted natural language into sign language gesture information,

[1016] A device that displays converted sign language motion information as an animation,

[1017] The information processing system according to claim 1, including the following:

[1018] (Claim 3)

[1019] The information processing system according to claim 1, further comprising a learning device for learning the individual characteristics of users and improving conversion accuracy.

[1020] "Application Example 1"

[1021] (Claim 1)

[1022] A means of acquiring video data for recognizing sign language,

[1023] A method for extracting the characteristics of sign language from acquired video data,

[1024] A means of analyzing extracted features and converting them into corresponding natural language,

[1025] A means for converting the converted natural language data into speech and outputting it via a speech output device,

[1026] A means of acquiring audio data and converting it into natural language,

[1027] A means of converting the converted natural language into sign language motion data,

[1028] A means for displaying converted sign language motion data on a visual display device,

[1029] A means of learning user characteristics using feedback information and improving conversion accuracy,

[1030] A system that includes this.

[1031] (Claim 2)

[1032] The system according to claim 1, further comprising means for optimizing the transformation model based on feedback.

[1033] (Claim 3)

[1034] The system according to claim 1, implemented as an application installed on a device.

[1035] "Example 2 of combining an emotion engine"

[1036] (Claim 1)

[1037] In addition to sign language, a means of acquiring video data to recognize the user's emotions,

[1038] A means for extracting the characteristics of sign language and emotions based on the user's facial expressions from acquired video data,

[1039] A method for analyzing the characteristics and emotions of extracted sign language and converting them into corresponding natural language while taking emotions into consideration,

[1040] A means for outputting the converted natural language data as speech,

[1041] A system that includes this.

[1042] (Claim 2)

[1043] A means of acquiring audio data, converting it into natural language, and then converting it into sign language motion data with appropriate emotions,

[1044] A means of outputting the converted sign language motion data as an animation,

[1045] The system according to claim 1, including the following:

[1046] (Claim 3)

[1047] The system according to claim 1, further comprising a learning means for improving the accuracy of recognizing individual user characteristics and emotions.

[1048] "Application example 2 when combining with an emotional engine"

[1049] (Claim 1)

[1050] Means of obtaining information,

[1051] A method for extracting the characteristics of sign language from acquired information,

[1052] The extracted features are analyzed to provide a means of recognizing emotions,

[1053] A means of converting and reflecting emotions into the corresponding natural language,

[1054] A means of outputting the converted natural language data as speech,

[1055] A system that includes this.

[1056] (Claim 2)

[1057] A means of acquiring speech and converting it into natural language that takes emotions into account,

[1058] A means of converting the converted natural language into sign language motion data,

[1059] A means of outputting converted sign language motion data as an animation,

[1060] The system according to claim 1, including the following:

[1061] (Claim 3)

[1062] The system according to claim 1, further comprising a learning means for learning the individual emotional patterns of users and improving conversion accuracy. [Explanation of symbols]

[1063] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring video data for recognizing sign language, A method for extracting the characteristics of sign language from acquired video data, A means of analyzing extracted features and converting them into corresponding natural language, A means of outputting the converted natural language data, A system that includes this.

2. A means of acquiring audio data and converting it into natural language, A means of converting the converted natural language into sign language motion data, A means for outputting converted sign language motion data, The system according to claim 1, including the following:

3. The system according to claim 1, further comprising a learning means for learning the individual characteristics of users and improving conversion accuracy.

Images