system

Abstract

Provide a system. [Solution] means for acquiring a video of a user's actions while using sign language; A means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame; a means for inputting the analysis results into a sign language recognition model to generate a corresponding string of characters; means for displaying the generated string to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Sign language plays an important role in communication between the hearing impaired and the hearing-impaired, but only a limited number of people can understand and use sign language. This has led to a demand for systems that convert sign language into text, and vice versa. However, existing technologies have issues such as low sign language recognition accuracy and unnatural sign language motion generation. Furthermore, there are challenges, such as the difficulty of translating in real time. [Means for solving the problem]

[0005] The present invention provides a system including a means for capturing a video of a user's sign language movements, a means for dividing the captured video into frames and analyzing the hand shape and position of each frame, a means for inputting the analysis results into a sign language recognition model to generate a corresponding character string, and a means for displaying the generated character string to the user. The sign language recognition model utilizes a deep learning algorithm to achieve high recognition accuracy. The system also includes a means for converting character data into sign language motion data, a means for rendering the sign language motion data into a 3D animation format, and a means for displaying the rendered animation to the user. This means enables bidirectional translation in real time, enabling the generation of natural sign language motion.

[0006] A "signing user" is a person who actually performs the gestures and movements of sign language.

[0007] "Means for capturing as video" refers to a device or method for capturing the actions of a user performing sign language in video format.

[0008] "Frame division" is the process of cutting out the consecutive images that make up a video into individual still images (frames).

[0009] "Means for analyzing hand shape and position" refers to algorithms or technologies that analyze the shape and position of the hand within the frame and convert it into data.

[0010] A "sign language recognition model" is a trained model that analyzes sign language gestures and movements and converts them into specific strings of characters and meanings.

[0011] A "deep learning algorithm" is a type of artificial intelligence technology that uses large amounts of data to perform advanced pattern recognition and classification.

[0012] "Means for generating a character string" refers to an algorithm or method for generating corresponding text data from the analysis results.

[0013] "Means for displaying to the user" refers to a display device or technology that visually presents the generated text and sign language motion to the user.

[0014] "Sign language motion data" is digital data that represents sign language actions, and includes information such as hand movements, positions, and shapes.

[0015] "Means of rendering into 3D animation format" refers to the techniques and processes used to generate 3D animation based on sign language motion data.

[0016] "Rendered animation" refers to sign language motion that has been rendered into a 3D animation format, making it visually displayable. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[0039] Translating sign language into text

[0040] 1. Acquiring video data

[0041] When the user signs, the device uses its camera to capture video of the sign.

[0042] The captured video data is sent to a server in real time.

[0043] 2. Analysis of video data

[0044] The video received by the server is divided into frames.

[0045] The hand position and shape in each frame are analyzed using a computer vision algorithm (e.g., OpenCV).

[0046] 3. Sign Language Identification

[0047] The server inputs the analyzed data into a sign language recognition model (e.g., a deep learning model).

[0048] The sign language recognition model analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[0049] 4. Conversion to Character Data

[0050] The server converts the identified sign language into a corresponding string.

[0051] The converted string is sent to the terminal, which displays the characters to the user.

[0052] Example (translating sign language into text)

[0053] 1. The user signs "hello."

[0054] 2. The device captures the sign language with its camera and sends the video to the server.

[0055] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[0056] 4. The server uses the recognition model to determine that this motion is a "hello."

[0057] 5. The server generates the string "Hello" and sends it to the terminal.

[0058] 6. The terminal displays this string to the user.

[0059] Translating text into sign language

[0060] 1. Entering text data

[0061] The user inputs text data (for example, "Thank you") into the terminal.

[0062] In the case of a live voice, the audio is recorded using the device's microphone and the audio data is converted into text.

[0063] The text data is sent to the server.

[0064] 2. Sign Language Generation

[0065] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data.

[0066] Once the motion data is generated, the server renders it into a 3D animation format.

[0067] 3. Sending sign language animations

[0068] The server transmits the generated sign language animation data to the terminal.

[0069] Users can view sign language animations on their devices.

[0070] Example (translating text into sign language)

[0071] 1. The user types the word "thank you" into the terminal.

[0072] 2. The terminal sends the character data to the server.

[0073] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[0074] 4. The server renders this motion into a 3D animation and sends it back to the device.

[0075] 5. The device plays the animation and displays it to the user.

[0076] This allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motion, enabling smooth communication with the hearing impaired.

[0077] The processing flow will be explained below.

[0078] Translating sign language into text

[0079] Step 1:

[0080] The user signs a particular phrase or word.

[0081] Step 2:

[0082] The device uses a camera to record the user's sign language as a video, and the video data is sent to a server in real time.

[0083] Step 3:

[0084] The video data received by the server is divided into frames. This division is the process of converting the video into individual still images (frames).

[0085] Step 4:

[0086] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts data on the hand's shape, position, and movement characteristics.

[0087] Step 5:

[0088] The server feeds the parsed frames into a sign language recognition model (e.g., a deep learning model), which identifies sign language words and phrases based on hand movements and position information.

[0089] Step 6:

[0090] The server converts the signs that are identified with high confidence into corresponding strings of characters based on the output from the sign language recognition model.

[0091] Step 7:

[0092] The server sends the converted string data to the terminal.

[0093] Step 8:

[0094] The terminal displays the character string data received from the server to the user.

[0095] Translating text into sign language

[0096] Step 1:

[0097] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[0098] Step 2:

[0099] The terminal transmits the input character data to the server.

[0100] Step 3:

[0101] The server inputs the text data into a sign language generation algorithm, which converts the text data into sign language motion data.

[0102] Step 4:

[0103] The server renders 3D animation based on the generated sign language motion data. During this rendering process, the sign language motion is visually represented.

[0104] Step 5:

[0105] The server sends the rendered sign language animation data to the terminal.

[0106] Step 6:

[0107] The terminal plays the received sign language animation and displays it to the user.

[0108] This allows users to input sign language and convert it into text, and input text or voice and convert it into sign language in real time.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] Conventional sign language translation systems have had difficulty accurately recognizing sign language movements and converting them into text in real time. Furthermore, when converting text data into sign language motion, there was a lack of technology to generate natural-looking sign language animations. As a result, smooth communication with the hearing impaired was difficult.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes means for capturing user actions as video, means for dividing the captured video into frames and analyzing the hand shape and position in each frame, means for inputting the analysis results into a sign language recognition model and converting the sign language actions into corresponding character strings, and means for displaying the generated character strings to the user in real time, thereby enabling the sign language actions to be accurately and quickly converted into character strings and displayed to the user.

[0114] In this invention, the server further includes means for converting character data into sign language motion data, means for rendering the sign language motion data into a 3D animation format, and means for displaying the rendered animation to the user, thereby enabling input character data to be displayed as natural sign language animation.

[0115] "User" refers to a person who uses sign language or inputs character data.

[0116] "Terminal" refers to a device that includes a camera that captures video of the user signing and a device that inputs text data.

[0117] "Server" refers to a computer system that receives and analyzes video and text data sent from a terminal, and performs sign language recognition and generates 3D animation.

[0118] "Video data" refers to video information captured by a camera of a user's sign language actions.

[0119] A "frame" refers to each still image that makes up video data.

[0120] "Computer vision algorithm" refers to image processing technology for analyzing the position and shape of hands from video data.

[0121] "Sign language recognition model" refers to a deep learning model for identifying sign language words and phrases based on analyzed hand movements.

[0122] "Character string" refers to text information in which sign language actions are converted into corresponding characters.

[0123] "Sign language motion data" refers to data that expresses sign language movements and is generated based on character data.

[0124] "3D animation format" refers to a format that expresses sign language motion data as three-dimensional movement.

[0125] "Real-time" means that the system processes and converts data instantly and displays the results without delay.

[0126] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[0127] Translating sign language into text

[0128] When a user signs, the device uses a camera to capture video of the sign language. Specifically, a camera installed on a smartphone or PC is used. The captured video data is sent to a server in real time. The server divides the received video into frames and analyzes the position and shape of the hands in each frame. This analysis is performed using a computer vision algorithm such as OpenCV. The analyzed data is input into a sign language recognition model. A deep learning model built using TENSORFLOW (registered trademark) or PyTorch is applied as the sign language recognition model. The recognition model analyzes the hand movements in each frame and identifies the corresponding sign language word or phrase. The identified sign language is converted into the corresponding string of characters, and the converted string of characters is sent to the device. The device then displays the characters to the user.

[0129] For example, if a user signs "hello," the device captures the sign with a camera and sends the video to the server. The server divides the video into frames and recognizes the shape and position of the hand. The server then uses a recognition model to determine that this movement is "hello." The server generates the string "hello" and sends it to the device. The device then displays this string to the user.

[0130] Translating text into sign language

[0131] The user inputs text data (for example, "Thank you") into the device. If the input is human voice, the audio is recorded using the device's microphone, and the audio data is converted into text using a method such as the Google (registered trademark) Cloud Speech-to-Text API. This text data is sent to the server. The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data. The generated motion data is rendered in a 3D animation format. Unity or Blender is used for rendering. The generated sign language animation data is sent to the device, and the user can view the sign language animation on the device.

[0132] For example, if a user types the character "Thank you" into a device, the device sends the character data to the server. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "Thank you." The server renders this motion into a 3D animation and sends it back to the device. The device plays the animation and displays it to the user.

[0133] This system allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motions, enabling smooth communication with the hearing impaired.

[0134] Prompt Sentence Examples

[0135] "Please explain how this system translates sign language into text in real time and simultaneously renders sign language animations in multiple dimensions."

[0136] Hardware and software used

[0137] Hardware: smartphone, computer, camera, microphone

[0138] Software: OpenCV, TensorFlow, PyTorch, Google Cloud Speech-to-Text API, Unity, Blender

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] Acquiring video data

[0142] Input: User's sign language actions

[0143] The device captures the user's sign language movements with a camera, which can be a built-in camera on a smartphone or PC, and transmits the captured video data to a server in real time.

[0144] Step 2:

[0145] Video data analysis

[0146] Input: Video data from the device

[0147] Output: Hand position and shape data per frame

[0148] The server splits the received video into frames using a library such as FFmpeg. The position and shape of the hand in each frame is analyzed using a computer vision algorithm (e.g., OpenCV). For example, the position of the hand can be identified by detecting a specific color area within the frame.

[0149] Step 3:

[0150] Sign Language Identification

[0151] Input: Parsed frame data

[0152] Output: A string corresponding to the sign language action

[0153] The server inputs the parsed data into a sign language recognition model, which uses deep learning models built with TensorFlow and PyTorch. The model analyzes the hand movements in each frame and identifies which sign language words or phrases they correspond to. For example, a series of hand movements may have a pattern that corresponds to a specific sign language word.

[0154] Step 4:

[0155] Conversion to character data

[0156] Input: Recognition results of sign language recognition model

[0157] Output: The corresponding string

[0158] The server converts the sign language actions identified by the sign language recognition model into a corresponding string. For example, if the hand movement is identified as corresponding to "hello," the server generates the string "hello." The server then transmits the generated string to the terminal.

[0159] Step 5:

[0160] Displaying a string

[0161] Input: The string sent from the server

[0162] Output: The string the user sees on the terminal

[0163] The terminal displays the text sent from the server on the screen in real time using a standard text drawing library.

[0164] Step 6:

[0165] Entering character data

[0166] Input: Character data entered by the user.

[0167] The user inputs text data (for example, "Thank you") into the device. In the case of voice input, the voice is recorded with the device's microphone and converted into text using the Google Cloud Speech-to-Text API or similar. The text data is then sent to the server.

[0168] Step 7:

[0169] Generating sign language motion data

[0170] Input: Character data

[0171] Output: Sign language motion data

[0172] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data, which expresses the sign language movements in detail.

[0173] Step 8:

[0174] 3D animation rendering

[0175] Input: Sign language motion data

[0176] Output: 3D animation data

[0177] The server renders the sign language motion data into a 3D animation format using Unity or Blender to visually represent the motion data.

[0178] Step 9:

[0179] Playing and viewing animations

[0180] Input: 3D animation data

[0181] Output: Sign language animation that the user sees on their device

[0182] The server sends the generated 3D animation data to the device, which then plays the animation and displays it to the user, allowing the user to see the sign language through the animation.

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Conventional sign language recognition systems can convert sign language into text and vice versa, but they lack the support functions to facilitate smooth face-to-face communication in brick-and-mortar stores, etc. In particular, there is a demand for a system that uses smart glasses to convert between sign language and text in real time and display sign language motion as 3D animation, enabling smooth communication with the hearing impaired.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes: means for capturing a video of a user's sign language movements; means for dividing the captured video into frames and analyzing the hand shape and position of each frame; means for inputting the analysis results into a sign language recognition model and generating a corresponding character string; means for displaying the generated character string on a displayable device; means for displaying the generated character string on smart glasses; means for converting character data into sign language motion data; means for rendering the sign language motion data into a 3D animation format; and means for displaying the rendered animation on a displayable device. This enables real-time conversion between sign language and characters using smart glasses in face-to-face communication such as in brick-and-mortar stores, enabling smooth communication with hearing-impaired people.

[0188] "Means for capturing the actions of a user performing sign language as video" refers to a device that uses a camera or other sensor to record the sign language movements of a user as video data.

[0189] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a system that divides the recorded video into frames and uses algorithms and computer vision technology to analyze the position and shape of the hand in each frame.

[0190] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding character string" refers to the process of inputting the analysis results of hand position and shape into a sign language recognition model and generating a corresponding character string as a result. This sign language recognition model utilizes technologies such as machine learning and deep learning.

[0191] The "means for presenting the generated character string on a displayable device" refers to a method for outputting the generated character string to a display device such as a smartphone, tablet, or computer display.

[0192] The "means for displaying the generated character string on the smart glasses" refers to a device that displays the generated character string on the display of the smart glasses in real time. When a user wears the smart glasses, they can simultaneously view the real world and the display.

[0193] "Means for converting character data into sign language motion data" refers to an algorithm or computer program for converting input character data into sign language motion, thereby converting the character information into dynamic sign language expressions.

[0194] "Means for rendering sign language motion data into a 3D animation format" refers to a process or system that generates 3D animation from sign language motion data. This rendering creates a visually realistic 3D animation of the sign language motion.

[0195] A "means for presenting rendered animation on a display-capable device" refers to a method or apparatus for showing the generated 3D animation to a user on a smartphone, tablet, computer display, etc.

[0196] The present invention relates to a system for converting sign language into text in real time, and a system for converting text into sign language. This system has the function of acquiring and analyzing a user's sign language movements and converting them into text, and conversely, converting the text into sign language motion data and displaying it. Specific embodiments of the present invention will be described below.

[0197] Hardware and Software Configuration

[0198] This system is implemented mainly using the following hardware and software.

[0199] Hardware: Smart glasses (with camera and display capabilities), server, displayable devices (smartphones, tablets, computers, etc.)

[0200] software:

[0201] Computer vision algorithms: Image processing libraries such as OpenCV

[0202] Sign language recognition model library: HandSignRecognitionModel

[0203] A library for converting text to sign language motion: TextToSignModel

[0204] 3D animation rendering library: Render3DModel

[0205] Overview of data processing and calculation

[0206] 1. Acquiring video data

[0207] The server uses a camera to record the sign language actions performed by the user. The camera in the smart glasses captures the sign language in real time and sends the video data to the server.

[0208] 2. Analysis of video data

[0209] The acquired video data is split into frames on the server, and computer vision algorithms such as OpenCV are used to analyze the shape and position of the hand in each frame.

[0210] 3. Sign Language Identification

[0211] The analysis results of hand shape and position are input into a sign language recognition model (HandSignRecognitionModel), which uses a deep learning algorithm to analyze the hand movements in each frame and identify the corresponding sign language word or phrase.

[0212] 4. Conversion to Character Data

[0213] The sign language identified by the sign language recognition model is converted into the corresponding text on the server and sent to a display-capable device or smart glasses, which then display the text information on the display in real time.

[0214] 5. Converting character data into sign language motion data

[0215] The server inputs character data from the user into a TextToSignModel and generates corresponding sign language motion data, which is then rendered into a 3D animation format using a Render3DModel.

[0216] 6. Displaying Sign Language Animation

[0217] The generated 3D animation is sent from the server to smart glasses or other display-capable devices and displayed in real time.

[0218] Specific operation example

[0219] Example 1: Sign language to text conversion

[0220] 1. The user signs "hello."

[0221] 2. The camera in the smart glasses captures the sign language and sends the video to a server.

[0222] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[0223] 4. The server uses a sign language recognition model to identify this movement as "hello."

[0224] 5. The server generates the string "hello" and sends it to the smart glasses.

[0225] 6. The smart glasses will display this string in real time.

[0226] Example 2: Converting text to sign language

[0227] 1. The user types "thank you" into their smartphone.

[0228] 2. The entered character data is sent to the server.

[0229] 3. The server uses TextToSignModel to generate sign language motion data corresponding to "thank you."

[0230] 4. The server renders this motion into a 3D animation and sends it to the smart glasses.

[0231] 5. The smart glasses play the animation and display it to the user.

[0232] Prompt Sentence Examples

[0233] Examples of prompts for generative AI models include:

[0234] prompt:

[0235] A user signs "Hello." This video is captured by a camera and sent to a sign language recognition system. The system uses a sign language recognition model to analyze the movement and generate the corresponding text "Hello." This text should then be displayed on the smart glasses.

[0236] output:

[0237] The text "Hello" will appear on the smartglasses display.

[0238] This will enable real-time conversion between sign language and text using smart glasses during face-to-face communication in brick-and-mortar stores, enabling smooth communication with the hearing impaired.

[0239] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0240] Step 1:

[0241] The user signs.

[0242] Input: The user's sign language action.

[0243] Output: Video data captured by the camera.

[0244] Specific operation: The camera on the device (smart glasses) captures the user's sign language movements in real time and acquires them as video data.

[0245] Step 2:

[0246] The video data acquired by the terminal is sent to the server.

[0247] Input: Video data captured by the camera.

[0248] Output: Video data sent to the server.

[0249] Specific operation: The network connection is used to send the video data acquired by the device to the server in real time.

[0250] Step 3:

[0251] The server divides the video data into frames and analyzes the shape and position of the hand in each frame.

[0252] Input: Video data received by the server.

[0253] Output: Analysis results of hand shape and position for each frame.

[0254] How it works: The server divides the video data into frames and uses computer vision algorithms such as OpenCV to identify the shape and position of the hand for each frame.

[0255] Step 4:

[0256] The server inputs the analysis results into a sign language recognition model (HandSignRecognitionModel) and generates a corresponding string.

[0257] Input: Analysis results of hand shape and position.

[0258] Output: The corresponding string data.

[0259] How it works: The server inputs hand shape and position data into a sign language recognition model, then uses a deep learning algorithm to analyze the sign language and generate a corresponding string of characters.

[0260] Step 5:

[0261] The server sends the generated string to a display-capable device or smart glasses.

[0262] Input: String data generated by a sign language recognition model.

[0263] Output: The string to be displayed on the smart glasses.

[0264] Specific operation: The server sends the generated string data to the smart glasses via the network and displays it in real time.

[0265] Step 6:

[0266] The user inputs character data.

[0267] Input: Character data entered by the user.

[0268] Output: Character data entered into the terminal.

[0269] Specific operation: The user inputs text data (e.g., "Thank you") using a device such as a smartphone or tablet.

[0270] Step 7:

[0271] The terminal transmits the input character data to the server.

[0272] Input: Character data entered by the user into a terminal.

[0273] Output: Character data sent to the server.

[0274] Specific operation: The terminal transmits the input character data to the server in real time via the network.

[0275] Step 8:

[0276] The server converts the character data into sign language motion data.

[0277] Input: Character data received by the server.

[0278] Output: The corresponding sign language motion data.

[0279] Specific operation: The server uses TextToSignModel to generate sign language motion data from character data.

[0280] Step 9:

[0281] The server renders the sign language motion data into a 3D animation format.

[0282] Input: Generated sign language motion data.

[0283] Output: Sign language motion in 3D animation format.

[0284] Specific operation: The server uses Render3DModel to convert sign language motion data into 3D animation.

[0285] Step 10:

[0286] The server sends the rendered 3D animation to a device capable of displaying it.

[0287] Input: Sign language motion in 3D animation format.

[0288] Output: Sign language animation displayed on smart glasses.

[0289] Specific operation: The server sends the generated 3D animation to the smart glasses via the network and displays it in real time.

[0290] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0291] This invention adds an emotion engine that recognizes the user's emotions to systems that translate sign language into text and systems that translate text into sign language, thereby improving the accuracy of sign language translation and realizing natural communication that takes emotions into account.

[0292] Translating sign language into text

[0293] 1. Acquiring video data

[0294] When the user signs, the device uses its camera to capture video of the sign.

[0295] The video data captured by the device is sent to the server in real time.

[0296] 2. Analysis of video data

[0297] The video data received by the server is divided into frames, and the position and shape of the hand in each frame is analyzed using a computer vision algorithm.

[0298] The server extracts the user's facial expressions from the video frames and recognizes the user's emotions using an emotion engine that uses facial recognition algorithms and natural language processing algorithms.

[0299] 3. Sign Language Identification

[0300] The server inputs the analysis results into a sign language recognition model (e.g., a deep learning model), which analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[0301] The system adjusts responses by combining recognized sign language with emotions inferred from the user's facial expressions.

[0302] 4. Conversion to Character Data

[0303] The server converts sign language into corresponding text, selecting an appropriate expression that reflects the user's emotions.

[0304] The server sends the converted string to the terminal and displays it to the user.

[0305] Example (translating sign language into text)

[0306] 1. The user smiles and signs "hello."

[0307] 2. The device uses its camera to capture sign language and facial expressions and sends the video to the server.

[0308] 3. The server divides the video into frames and analyzes the shape and position of the hands and facial expressions.

[0309] 4. The server uses the sign language recognition model to identify "hello" and the emotion engine recognizes a smile.

[0310] 5. The server generates the string "hello" and displays it in a friendly way.

[0311] 6. The terminal displays this string to the user.

[0312] Translating text into sign language

[0313] 1. Entering text data

[0314] The user inputs text data (e.g., "Thank you") into the terminal.

[0315] The terminal transmits the input character data to the server.

[0316] 2. Sign Language Generation

[0317] The server inputs the character data into a sign language generation algorithm to generate corresponding sign language motion data.

[0318] The server uses an emotion engine to predict facial expressions from input text and adjusts motion data based on that information.

[0319] 3. Sending sign language animations

[0320] The server renders the generated sign language motion data into a 3D animation format.

[0321] The server sends the rendered sign language animation data to the terminal.

[0322] 4. Displaying Sign Language Animation

[0323] The terminal plays back the received animation to the user and displays it as sign language motion that includes emotion.

[0324] Example (translating text into sign language)

[0325] 1. The user types the word "thank you" into the terminal.

[0326] 2. The terminal sends the character data to the server.

[0327] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[0328] 4. The server uses an emotion engine to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[0329] 5. The server renders this motion into a 3D animation and sends it back to the device.

[0330] 6. The device plays the animation and displays the sign language animation that reflects the emotion to the user.

[0331] This allows users to communicate naturally in real time, and conversion between sign language and text is performed in a way that includes emotional information.

[0332] The processing flow will be explained below.

[0333] Translating sign language into text

[0334] Step 1:

[0335] The user signs, using natural facial expressions and eye contact along with hand movements.

[0336] Step 2:

[0337] The device uses a camera to capture video of the user's sign language and facial expressions, and sends the captured video data to a server in real time.

[0338] Step 3:

[0339] The video data received by the server is divided into frames, which is a process of converting the video file into still image frames.

[0340] Step 4:

[0341] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts features of the hand shape, position, and movement.

[0342] Step 5:

[0343] The server analyzes the user's face in the video frames and uses facial recognition algorithms to recognize facial expressions, as well as natural language processing algorithms to infer emotions from facial expressions.

[0344] Step 6:

[0345] The server feeds the analyzed frames into a sign language recognition model (e.g., a deep learning model), which identifies the sign language words or phrases that correspond to the movements in each frame.

[0346] Step 7:

[0347] The server converts the sign language into the corresponding text based on the output of the sign language recognition model. At this time, the server adjusts the expression of the text taking into account the user's emotion estimated by the emotion engine.

[0348] Step 8:

[0349] The server sends the converted string data to the terminal.

[0350] Step 9:

[0351] The terminal displays the character string data received from the server to the user in an appropriate format that reflects the user's emotions.

[0352] Translating text into sign language

[0353] Step 1:

[0354] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[0355] Step 2:

[0356] The terminal sends the input or converted character data to the server.

[0357] Step 3:

[0358] The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data based on the text data.

[0359] Step 4:

[0360] The server uses an emotion engine to recognize the emotional tone from the input text data and voice data, and adjusts the sign language motion data based on the emotion estimated from the text data and voice data.

[0361] Step 5:

[0362] The server renders 3D animation based on the generated sign language motion data, which visualizes natural sign language motion that reflects emotional information.

[0363] Step 6:

[0364] The server sends the rendered sign language animation data to the terminal.

[0365] Step 7:

[0366] The terminal then plays back the received sign language animation and displays it to the user. This display is a natural sign language animation that reflects the user's emotions.

[0367] This allows users to easily translate between sign language and text in real time, incorporating emotions.

[0368] Example 2

[0369] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0370] When communicating between sign language users and other people, there is a need for a system that can accurately understand sign language and express emotions in real time. However, existing sign language translation systems have low sign language recognition accuracy and do not take into account the user's emotional expression, making natural communication difficult. To solve this problem, a system is needed that improves the accuracy of sign language recognition and translates the user's emotions.

[0371] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0372] In this invention, the server includes means for capturing a video of a user's sign language movements, means for dividing the captured video into frames and analyzing the hand shape and position of each frame, means for extracting the user's facial expressions from the captured video frames and recognizing their emotions, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, and means for displaying the generated character string to the user. This enables highly accurate recognition of the user's sign language and natural translation that reflects the user's emotions.

[0373] "Means for capturing video of the actions of a sign language user" refers to devices such as cameras and sensors for recording the sign language actions of a sign language user, and includes technology for collecting those actions as digital video data.

[0374] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a software algorithm that breaks down the recorded video data into individual frames and identifies the position and shape of the user's hand in each frame. Specifically, it utilizes computer vision technology.

[0375] "Means for extracting a user's facial expressions from captured video frames and recognizing emotions" refers to a facial recognition algorithm and emotion analysis engine that analyzes facial features contained in video frames and grasps the user's emotional state.

[0376] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to the process of inputting the analyzed hand movement and shape data into a sign language recognition model (e.g., a model using a deep learning algorithm) and outputting the corresponding string of characters as a result.

[0377] "Means for displaying the generated character string to the user" refers to a device or software component for visually displaying the character string sent from the server to the user. Specifically, this includes the display of a smartphone or a dedicated display device.

[0378] "Means for converting character data into sign language motion data" refers to a software algorithm for converting input character string data into corresponding sign language motion data. Specifically, it includes a sign language dictionary database and a motion generation model.

[0379] "Means for rendering sign language motion data into a 3D animation format" refers to technology for rendering sign language motion data as 3D computer graphics to generate visually realistic sign language animation. Specifically, this refers to the use of 3D modeling software and rendering engines.

[0380] "Means for displaying the rendered animation to the user" refers to a video player component or display device that plays the generated 3D animation on the user's device, including a computer or mobile device display.

[0381] The present invention adds an emotion engine that recognizes the user's emotions to a system that translates sign language into text and a system that translates text into sign language. This improves the accuracy of sign language translation and realizes natural communication that reflects emotions. Specific embodiments of the present invention are described below.

[0382] Translating sign language into text

[0383] 1. Acquiring video data

[0384] The user signs.

[0385] The device (smartphone or camera-equipped device) records the sign language as a video, capturing the video in real time using the device's built-in camera.

[0386] The video data captured by the device is encoded in real time and sent to a server via the Internet using the RTMP protocol or WebRTC.

[0387] 2. Analysis of video data

[0388] The video data received by the server is divided into frames. This process uses a library such as FFmpeg.

[0389] The server analyzes each frame and uses OpenCV to identify the position and shape of the hand, using algorithms such as Edge Detection and Contour Detection.

[0390] The server uses a facial recognition algorithm (dlib, FaceNet, etc.) to extract the user's facial expressions from video frames, and then uses an emotion engine that utilizes natural language processing algorithms such as BERT to recognize the user's emotions.

[0391] 3. Sign Language Identification

[0392] The server inputs the analysis results into a sign language recognition model (a deep learning model built with TensorFlow or PyTorch), which analyzes the hand movements in each frame and identifies the corresponding sign language words and phrases.

[0393] The server combines the sign language recognition results with the emotions inferred by the emotion engine and adjusts the response.

[0394] 4. Conversion to Character Data

[0395] The server converts the sign language into a corresponding string of characters, choosing an appropriate expression that reflects the user's emotions. For example, for the sign "hello," the server converts it into "hello!", taking into account the smile.

[0396] The server encodes the generated character data and sends it to the terminal via the Internet.

[0397] The terminal decodes the received character data and displays it on the user's screen. Specifically, this is the operation of setting the character string in a GUI component.

[0398] Example (translating sign language into text)

[0399] The user smiles and signs "hello."

[0400] The device uses a camera to capture the sign language and facial expressions, and then sends the video to a server via the RTMP protocol.

[0401] The server splits the video into frames, analyzes hand shape and position using OpenCV, and analyzes facial expressions using dlib.

[0402] The server identifies "hello" using a TensorFlow model, and the emotion engine recognizes a smile using BERT.

[0403] The server generates the string "Hello!" and sends it to the terminal.

[0404] The terminal displays this string to the user.

[0405] Translating text into sign language

[0406] 1. Entering text data

[0407] The user enters text data (e.g., "Thank you") into the terminal, using a touchscreen or keyboard.

[0408] The device sends the input character data to the server via HTTP or WebSocket.

[0409] 2. Sign Language Generation

[0410] The server receives the character data and inputs it into a sign language generation algorithm (e.g., a deep learning model), which searches for corresponding sign language motion data from a character-to-sign language dictionary database.

[0411] The server uses an emotion engine to predict facial expressions from the input text and adjusts the sign language motion data based on that information. Specifically, it adjusts the movement and speed of the motion data to correspond to the emotion.

[0412] 3. Sending sign language animations

[0413] The server renders the generated sign language motion data using Blender or other 3D rendering software, and animates the 3D model of the sign language character.

[0414] The server encodes the 3D animation data and sends it to the device via the Internet.

[0415] 4. Displaying Sign Language Animation

[0416] The device decodes the received animation and plays it on the user's screen. Specifically, this is done by playing it using the video player component in the app.

[0417] The device displays emotional sign language motions to the user in 3D animation format.

[0418] Example (translating text into sign language)

[0419] The user types the word "thank you" into the device, for example, using the keyboard or touch screen of a smartphone.

[0420] The terminal sends the character data to the server via HTTP.

[0421] The server uses a sign language generation algorithm (TensorFlow model) to generate sign language motion data corresponding to "thank you."

[0422] The server uses an emotion engine (using the BERT model) to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[0423] The server uses Blender to render this motion data into a 3D animation, encodes it, and then sends it to the device.

[0424] The device decodes the animation and uses a video player to play the emotion-reflecting sign language animation on the user's screen.

[0425] Example prompts to input to the generative AI model

[0426] 1. Translating sign language to text: "When a user says 'hello' in sign language, analyze their hand movements and facial expressions and generate an appropriate string of text based on their emotions."

[0427] 2. Example of translating text into sign language: "Generate motion data to express the word 'thank you' in sign language, and create a 3D animation that reflects the emotion of gratitude."

[0428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0429] Translating sign language into text

[0430] Step 1:

[0431] The user signs

[0432] Input: User's sign language actions

[0433] Specific action: The user signs to the camera, for example, signing "hello."

[0434] Output: Video data including the user's sign language

[0435] Step 2:

[0436] The device captures video and sends it to the server in real time.

[0437] Input: Video data containing the user's sign language

[0438] Specific operation: The system uses the device's camera to capture sign language video in real time, encodes the video data using the RTMP protocol, and then transmits the encoded video data to a server via the Internet.

[0439] Output: Video data sent to the server

[0440] Step 3:

[0441] The server divides the video data into frames and analyzes the shape and position of the hand.

[0442] Input: Video data

[0443] How it works: The server splits the video data into frames using the FFmpeg library. For each frame, it analyzes the hand shape and position using the Edge Detection and Contour Detection algorithms using OpenCV.

[0444] Output: Analysis results of hand shape and position

[0445] Step 4:

[0446] The server extracts facial expressions from video frames and recognizes emotions.

[0447] Input: Video data

[0448] Specific operation: The server uses dlib and FaceNet to detect facial features from frames, and uses BERT, a natural language processing algorithm, to recognize the user's emotions from the detected facial features.

[0449] Output: User emotion data

[0450] Step 5:

[0451] The server inputs the analysis results into a sign language recognition model and generates the corresponding string.

[0452] Input: Analysis results of hand shape and position, user emotion data

[0453] How it works: The server inputs the analysis results into a deep learning model built with TensorFlow or PyTorch, converts the sign language gestures into corresponding text, and uses the emotion data to select the appropriate expression method as needed.

[0454] Output: The corresponding string

[0455] Step 6:

[0456] The server sends the generated string to the terminal, which displays the string to the user.

[0457] Input: The corresponding string

[0458] Specific operation: The server encodes the generated string and sends it to the terminal via the Internet. The terminal decodes the received string and displays it in the GUI component.

[0459] Output: The string displayed on the user's screen

[0460] Translating text into sign language

[0461] Step 1:

[0462] The user enters character data

[0463] Input: Text data (e.g. "Thank you")

[0464] Specific Action: A user types text using the keyboard or touchscreen of a smartphone or tablet.

[0465] Output: Character data entered

[0466] Step 2:

[0467] The terminal sends the input character data to the server.

[0468] Input: Character data entered

[0469] Specific operation: The terminal sends character data to the server using HTTP or WebSocket.

[0470] Output: Character data sent to the server

[0471] Step 3:

[0472] The server converts the text data into sign language motion data.

[0473] Input: Character data

[0474] How it works: The server inputs character data into a sign language generation algorithm (e.g., a TensorFlow model) to generate sign language motion data. It then uses an emotion engine to infer emotions from the character data and adjusts the motion data based on the results.

[0475] Output: Sign language motion data

[0476] Step 4:

[0477] The server renders the sign language motion data into a 3D animation format.

[0478] Input: Sign language motion data

[0479] Specific operation: The server uses 3D rendering software such as Blender to render the sign language motion data into 3D animation.

[0480] Output: 3D animation data

[0481] Step 5:

[0482] The server sends 3D animation data to the terminal, and the terminal displays the animation to the user.

[0483] Input: 3D animation data

[0484] Specific operation: The server encodes 3D animation data and sends it to the device via the Internet. The device decodes the received animation data and plays it using the video player component.

[0485] Output: The animation played on the user's screen

[0486] (Application example 2)

[0487] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0488] Conventional sign language translation systems simply convert sign language into text, making it difficult to achieve natural communication that reflects the user's emotions. Furthermore, they lacked the ability to output text generated from sign language as speech, or the ability to convert text into sign language animation. This made it difficult for hearing-impaired people and people who use sign language to communicate smoothly on public transportation, such as self-driving vehicles.

[0489] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a video of a user's actions using sign language, means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, means for extracting the user's facial expression from the video data and recognizing the emotion using an emotion engine, means for adjusting the character string to reflect the user's emotion, means for outputting the generated character string as audio, means for converting character data into sign language motion data, means for rendering the sign language motion data in a 3D animation format, and means for reflecting emotion in the rendered animation using the emotion engine. This enables users to communicate naturally and emotionally even inside an autonomous vehicle.

[0490] 1. "Means for capturing the actions of a user using sign language as video" refers to a device or system that records the actions of a user using sign language in video format using an image capture device such as a camera.

[0491] 2. "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to an algorithm or software that breaks down video data into individual frames and analyzes the shape and position of the user's hand within each frame.

[0492] 3. "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to a process or system that inputs the analyzed hand movement data into a sign language recognition model and generates a corresponding string of characters as a result.

[0493] 4. "Means for displaying the generated character string to the user" refers to a device or system that outputs the generated character data to a display or other display device and presents it in a form that the user can visually confirm.

[0494] 5. "Means for extracting a user's facial expressions from video data and recognizing emotions using an emotion engine" means a process or system that extracts a user's facial expressions from video data and identifies the user's emotional state using machine learning or artificial intelligence algorithms.

[0495] 6. "Means for adjusting text to reflect user emotions" refers to an algorithm or process that appropriately changes the content and expression of generated text based on recognized user emotional information.

[0496] 7. "Means for audibly outputting the generated character string" refers to a device or system that uses speech synthesis technology to play back the generated character data as speech.

[0497] 8. "Means for converting character data into sign language motion data" means an algorithm or software that converts input character information into corresponding sign language motion data.

[0498] 9. "Means for rendering sign language motion data into a 3D animation format" means a process or system that renders the generated sign language motion data as a three-dimensional animation.

[0499] 10. "Means for reflecting emotions in rendered animation using an emotion engine" means a system or algorithm that applies an emotion engine to generated 3D animation and reflects emotions in the character's movements and expressions.

[0500] This invention relates to a system that enables users who use sign language to communicate naturally and emotionally within an autonomous vehicle. This system captures sign language movements as video and generates text, audio, and sign language animations based on the analysis results, thereby supporting comfortable communication for users.

[0501] System Program

[0502] The system uses the following hardware and software:

[0503] Hardware: In-vehicle camera, high-resolution display, on-board computer, audio output device

[0504] Software: Computer vision libraries (e.g., OpenCV), deep learning frameworks (e.g., TensorFlow, PyTorch), natural language processing libraries (e.g., NLTK, Transformers), emotion recognition engines

[0505] System action

[0506] 1. Acquiring video data:

[0507] The in-vehicle camera captures the user's sign language in real time as video, which is then broken down into frames and the shape and position of the hand in each frame are analyzed.

[0508] 2. Sign Language Analysis and Recognition:

[0509] The analyzed data is input into a sign language recognition model (deep learning algorithm) to generate a corresponding string of characters. At the same time, the user's facial expressions are extracted from the video data, and an emotion engine recognizes the user's emotions.

[0510] 3. Creating and displaying strings:

[0511] The system adjusts the text based on the recognized sign language and emotional information, and the resulting text is displayed on the vehicle's display. The text is also output as speech using speech synthesis technology.

[0512] 4. Generating Sign Language Animation:

[0513] Conversely, text data entered by the user is converted into sign language motion data, which is then rendered into a 3D animation format and emotion-based using an emotion recognition engine. The final rendered animation is then displayed on the screen.

[0514] Specific examples

[0515] Translating sign language into text and speech

[0516] 1. A user signs "thank you" inside a self-driving vehicle.

[0517] 2. Cameras inside the vehicle capture sign language in real time and transmit the data to an onboard computer.

[0518] 3. The on-board computer divides the video into frames and analyzes hand movements and the user's facial expressions.

[0519] 4. Based on the analysis results, the sign language recognition model and emotion engine generate the string "Thank you" and emotional information.

[0520] 5. "Thank you" will appear on the display and be output as audio at the same time.

[0521] Translating text into sign language animation

[0522] 1. The user types "Help me" into the display.

[0523] 2. The on-board computer converts the text data into sign language motion data, and the emotion engine recognizes and reflects the emotion of "urgency."

[0524] 3. The generated sign language motion data is rendered as a 3D animation and displayed on the screen.

[0525] Prompt Sentence Examples

[0526] Translate sign language to text: A user signs "Where is my house?" You can see confusion in the user's face.

[0527] Information from sign language:

[0528] Transcription: Where is my house?

[0529] Emotion: Confused

[0530] Translate text into sign language: The user has typed "Thank you." Reflect the sentiment of gratitude with the emotion engine.

[0531] Generated sign language motion:

[0532] Sign Language: Thank you

[0533] Emotion: Gratitude

[0534] This will enable users to have natural and emotionally rich communication even inside an autonomous vehicle.

[0535] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0536] Step 1:

[0537] Acquiring video data using a camera: When a user signs, the device's in-car camera captures the action in real time. This captures video data including the user's hand movements and facial expressions. The input is "video data including the user's sign language and facial expressions," and the output is "video data acquired in real time."

[0538] Step 2:

[0539] Video data frame division and analysis: The server receives the acquired video data and divides the video into frames. The shape and position of the hand in each frame are analyzed using a computer vision library (e.g., OpenCV). The input is "video data acquired in real time," and the output is "hand shape and position data for each frame."

[0540] Step 3:

[0541] Sign language recognition: The server inputs the analyzed frame data into a sign language recognition model (e.g., a deep learning algorithm using TensorFlow or PyTorch) to identify the corresponding sign language words or phrases. The input is the hand shape and position data for each frame, and the output is the identified sign language string.

[0542] Step 4:

[0543] Facial expression analysis and emotion recognition: The server extracts the user's facial expressions from the video data and recognizes the user's emotions using a natural language processing algorithm and an emotion engine. The input is "video data acquired in real time" and the output is "user's emotional information."

[0544] Step 5:

[0545] Generating and displaying text and audio: The server adjusts the text based on the recognized sign language text and emotional information, and displays it on the vehicle's display. The text is also output as audio using a speech synthesis engine. The input is the "identified sign language text and the user's emotional information," and the output is "text and audio data displayed to the user."

[0546] Step 6:

[0547] Conversion of character data into sign language motion data: The server receives character data entered by the user and converts it into corresponding sign language motion data. The input is "character data entered by the user" and the output is "sign language motion data."

[0548] Step 7:

[0549] Rendering sign language motion data and reflecting emotions: The server renders the sign language motion data into 3D animation format and reflects emotional information using an emotion engine. The input is "sign language motion data" and "emotional information," and the output is "emotionally reflected sign language 3D animation."

[0550] Step 8:

[0551] Displaying sign language animation: The device displays the rendered sign language 3D animation on the display, providing a visual experience to the user. The input is "emotion-reflecting sign language 3D animation," and the output is "sign language 3D animation displayed to the user."

[0552] Through these steps, users can communicate naturally and emotively using sign language inside a self-driving vehicle.

[0553] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0554] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0555] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0556] [Second embodiment]

[0557] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0558] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0559] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0560] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0561] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0562] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0563] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0564] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0565] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0566] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0567] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0568] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0569] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[0570] Translating sign language into text

[0571] 1. Acquiring video data

[0572] When the user signs, the device uses its camera to capture video of the sign.

[0573] The captured video data is sent to a server in real time.

[0574] 2. Analysis of video data

[0575] The video received by the server is divided into frames.

[0576] The hand position and shape in each frame are analyzed using a computer vision algorithm (e.g., OpenCV).

[0577] 3. Sign Language Identification

[0578] The server inputs the analyzed data into a sign language recognition model (e.g., a deep learning model).

[0579] The sign language recognition model analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[0580] 4. Conversion to Character Data

[0581] The server converts the identified sign language into a corresponding string.

[0582] The converted string is sent to the terminal, which displays the characters to the user.

[0583] Example (translating sign language into text)

[0584] 1. The user signs "hello."

[0585] 2. The device captures the sign language with its camera and sends the video to the server.

[0586] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[0587] 4. The server uses the recognition model to determine that this motion is a "hello."

[0588] 5. The server generates the string "Hello" and sends it to the terminal.

[0589] 6. The terminal displays this string to the user.

[0590] Translating text into sign language

[0591] 1. Entering text data

[0592] The user inputs text data (for example, "Thank you") into the terminal.

[0593] In the case of a live voice, the audio is recorded using the device's microphone and the audio data is converted into text.

[0594] The text data is sent to the server.

[0595] 2. Sign Language Generation

[0596] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data.

[0597] Once the motion data is generated, the server renders it into a 3D animation format.

[0598] 3. Sending sign language animations

[0599] The server transmits the generated sign language animation data to the terminal.

[0600] Users can view sign language animations on their devices.

[0601] Example (translating text into sign language)

[0602] 1. The user types the word "thank you" into the terminal.

[0603] 2. The terminal sends the character data to the server.

[0604] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[0605] 4. The server renders this motion into a 3D animation and sends it back to the device.

[0606] 5. The device plays the animation and displays it to the user.

[0607] This allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motion, enabling smooth communication with the hearing impaired.

[0608] The processing flow will be explained below.

[0609] Translating sign language into text

[0610] Step 1:

[0611] The user signs a particular phrase or word.

[0612] Step 2:

[0613] The device uses a camera to record the user's sign language as a video, and the video data is sent to a server in real time.

[0614] Step 3:

[0615] The video data received by the server is divided into frames. This division is the process of converting the video into individual still images (frames).

[0616] Step 4:

[0617] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts data on the hand's shape, position, and movement characteristics.

[0618] Step 5:

[0619] The server feeds the parsed frames into a sign language recognition model (e.g., a deep learning model), which identifies sign language words and phrases based on hand movements and position information.

[0620] Step 6:

[0621] The server converts the signs that are identified with high confidence into corresponding strings of characters based on the output from the sign language recognition model.

[0622] Step 7:

[0623] The server sends the converted string data to the terminal.

[0624] Step 8:

[0625] The terminal displays the character string data received from the server to the user.

[0626] Translating text into sign language

[0627] Step 1:

[0628] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[0629] Step 2:

[0630] The terminal transmits the input character data to the server.

[0631] Step 3:

[0632] The server inputs the text data into a sign language generation algorithm, which converts the text data into sign language motion data.

[0633] Step 4:

[0634] The server renders 3D animation based on the generated sign language motion data. During this rendering process, the sign language motion is visually represented.

[0635] Step 5:

[0636] The server sends the rendered sign language animation data to the terminal.

[0637] Step 6:

[0638] The terminal plays the received sign language animation and displays it to the user.

[0639] This allows users to input sign language and convert it into text, and input text or voice and convert it into sign language in real time.

[0640] Example 1

[0641] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] Conventional sign language translation systems have had difficulty accurately recognizing sign language movements and converting them into text in real time. Furthermore, when converting text data into sign language motion, there was a lack of technology to generate natural-looking sign language animations. As a result, smooth communication with the hearing impaired was difficult.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0644] In this invention, the server includes means for capturing user actions as video, means for dividing the captured video into frames and analyzing the hand shape and position in each frame, means for inputting the analysis results into a sign language recognition model and converting the sign language actions into corresponding character strings, and means for displaying the generated character strings to the user in real time, thereby enabling the sign language actions to be accurately and quickly converted into character strings and displayed to the user.

[0645] In this invention, the server further includes means for converting character data into sign language motion data, means for rendering the sign language motion data into a 3D animation format, and means for displaying the rendered animation to the user, thereby enabling input character data to be displayed as natural sign language animation.

[0646] "User" refers to a person who uses sign language or inputs character data.

[0647] "Terminal" refers to a device that includes a camera that captures video of the user signing and a device that inputs text data.

[0648] "Server" refers to a computer system that receives and analyzes video and text data sent from a terminal, and performs sign language recognition and generates 3D animation.

[0649] "Video data" refers to video information captured by a camera of a user's sign language actions.

[0650] A "frame" refers to each still image that makes up video data.

[0651] "Computer vision algorithm" refers to image processing technology for analyzing the position and shape of hands from video data.

[0652] "Sign language recognition model" refers to a deep learning model for identifying sign language words and phrases based on analyzed hand movements.

[0653] "Character string" refers to text information in which sign language actions are converted into corresponding characters.

[0654] "Sign language motion data" refers to data that expresses sign language movements and is generated based on character data.

[0655] "3D animation format" refers to a format that expresses sign language motion data as three-dimensional movement.

[0656] "Real-time" means that the system processes and converts data instantly and displays the results without delay.

[0657] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[0658] Translating sign language into text

[0659] When a user signs, the device uses a camera to capture video of the sign language. Specifically, a camera installed on a smartphone or PC is used. The captured video data is sent to a server in real time. The server divides the received video into frames and analyzes the position and shape of the hands in each frame. This analysis is performed using computer vision algorithms such as OpenCV. The analyzed data is input into a sign language recognition model. A deep learning model built using TensorFlow or PyTorch is used as the sign language recognition model. The recognition model analyzes the hand movements in each frame and identifies the corresponding sign language word or phrase. The identified sign language is converted into the corresponding string of characters, and the converted string is sent to the device. The device then displays the characters to the user.

[0660] For example, if a user signs "hello," the device captures the sign with a camera and sends the video to the server. The server divides the video into frames and recognizes the shape and position of the hand. The server then uses a recognition model to determine that this movement is "hello." The server generates the string "hello" and sends it to the device. The device then displays this string to the user.

[0661] Translating text into sign language

[0662] The user inputs text data (for example, "Thank you") into the device. If the input is human voice, the audio is recorded using the device's microphone, and the audio data is converted into text using a method such as the Google Cloud Speech-to-Text API. This text data is sent to the server. The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data. The generated motion data is rendered in a 3D animation format. Rendering is done using Unity or Blender. The generated sign language animation data is sent to the device, and the user can view the sign language animation on the device.

[0663] For example, if a user types the character "Thank you" into a device, the device sends the character data to the server. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "Thank you." The server renders this motion into a 3D animation and sends it back to the device. The device plays the animation and displays it to the user.

[0664] This system allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motions, enabling smooth communication with the hearing impaired.

[0665] Prompt Sentence Examples

[0666] "Please explain how this system translates sign language into text in real time and simultaneously renders sign language animations in multiple dimensions."

[0667] Hardware and software used

[0668] Hardware: smartphone, computer, camera, microphone

[0669] Software: OpenCV, TensorFlow, PyTorch, Google Cloud Speech-to-Text API, Unity, Blender

[0670] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0671] Step 1:

[0672] Acquiring video data

[0673] Input: User's sign language actions

[0674] The device captures the user's sign language movements with a camera, which can be a built-in camera on a smartphone or PC, and transmits the captured video data to a server in real time.

[0675] Step 2:

[0676] Video data analysis

[0677] Input: Video data from the device

[0678] Output: Hand position and shape data per frame

[0679] The server splits the received video into frames using a library such as FFmpeg. The position and shape of the hand in each frame is analyzed using a computer vision algorithm (e.g., OpenCV). For example, the position of the hand can be identified by detecting a specific color area within the frame.

[0680] Step 3:

[0681] Sign Language Identification

[0682] Input: Parsed frame data

[0683] Output: A string corresponding to the sign language action

[0684] The server inputs the parsed data into a sign language recognition model, which uses deep learning models built with TensorFlow and PyTorch. The model analyzes the hand movements in each frame and identifies which sign language words or phrases they correspond to. For example, a series of hand movements may have a pattern that corresponds to a specific sign language word.

[0685] Step 4:

[0686] Conversion to character data

[0687] Input: Recognition results of sign language recognition model

[0688] Output: The corresponding string

[0689] The server converts the sign language actions identified by the sign language recognition model into a corresponding string. For example, if the hand movement is identified as corresponding to "hello," the server generates the string "hello." The server then transmits the generated string to the terminal.

[0690] Step 5:

[0691] Displaying a string

[0692] Input: The string sent from the server

[0693] Output: The string the user sees on the terminal

[0694] The terminal displays the text sent from the server on the screen in real time using a standard text drawing library.

[0695] Step 6:

[0696] Entering character data

[0697] Input: Character data entered by the user.

[0698] The user inputs text data (for example, "Thank you") into the device. In the case of voice input, the voice is recorded with the device's microphone and converted into text using the Google Cloud Speech-to-Text API or similar. The text data is then sent to the server.

[0699] Step 7:

[0700] Generating sign language motion data

[0701] Input: Character data

[0702] Output: Sign language motion data

[0703] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data, which expresses the sign language movements in detail.

[0704] Step 8:

[0705] 3D animation rendering

[0706] Input: Sign language motion data

[0707] Output: 3D animation data

[0708] The server renders the sign language motion data into a 3D animation format using Unity or Blender to visually represent the motion data.

[0709] Step 9:

[0710] Playing and viewing animations

[0711] Input: 3D animation data

[0712] Output: Sign language animation that the user sees on their device

[0713] The server sends the generated 3D animation data to the device, which then plays the animation and displays it to the user, allowing the user to see the sign language through the animation.

[0714] (Application example 1)

[0715] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0716] Conventional sign language recognition systems can convert sign language into text and vice versa, but they lack the support functions to facilitate smooth face-to-face communication in brick-and-mortar stores, etc. In particular, there is a demand for a system that uses smart glasses to convert between sign language and text in real time and display sign language motion as 3D animation, enabling smooth communication with the hearing impaired.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0718] In this invention, the server includes: means for capturing a video of a user's sign language movements; means for dividing the captured video into frames and analyzing the hand shape and position of each frame; means for inputting the analysis results into a sign language recognition model and generating a corresponding character string; means for displaying the generated character string on a displayable device; means for displaying the generated character string on smart glasses; means for converting character data into sign language motion data; means for rendering the sign language motion data into a 3D animation format; and means for displaying the rendered animation on a displayable device. This enables real-time conversion between sign language and characters using smart glasses in face-to-face communication such as in brick-and-mortar stores, enabling smooth communication with hearing-impaired people.

[0719] "Means for capturing the actions of a user performing sign language as video" refers to a device that uses a camera or other sensor to record the sign language movements of a user as video data.

[0720] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a system that divides the recorded video into frames and uses algorithms and computer vision technology to analyze the position and shape of the hand in each frame.

[0721] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding character string" refers to the process of inputting the analysis results of hand position and shape into a sign language recognition model and generating a corresponding character string as a result. This sign language recognition model utilizes technologies such as machine learning and deep learning.

[0722] The "means for presenting the generated character string on a displayable device" refers to a method for outputting the generated character string to a display device such as a smartphone, tablet, or computer display.

[0723] The "means for displaying the generated character string on the smart glasses" refers to a device that displays the generated character string on the display of the smart glasses in real time. When a user wears the smart glasses, they can simultaneously view the real world and the display.

[0724] "Means for converting character data into sign language motion data" refers to an algorithm or computer program for converting input character data into sign language motion, thereby converting the character information into dynamic sign language expressions.

[0725] "Means for rendering sign language motion data into a 3D animation format" refers to a process or system that generates 3D animation from sign language motion data. This rendering creates a visually realistic 3D animation of the sign language motion.

[0726] A "means for presenting rendered animation on a display-capable device" refers to a method or apparatus for showing the generated 3D animation to a user on a smartphone, tablet, computer display, etc.

[0727] The present invention relates to a system for converting sign language into text in real time, and a system for converting text into sign language. This system has the function of acquiring and analyzing a user's sign language movements and converting them into text, and conversely, converting the text into sign language motion data and displaying it. Specific embodiments of the present invention will be described below.

[0728] Hardware and Software Configuration

[0729] This system is implemented mainly using the following hardware and software.

[0730] Hardware: Smart glasses (with camera and display capabilities), server, displayable devices (smartphones, tablets, computers, etc.)

[0731] software:

[0732] Computer vision algorithms: Image processing libraries such as OpenCV

[0733] Sign language recognition model library: HandSignRecognitionModel

[0734] A library for converting text to sign language motion: TextToSignModel

[0735] 3D animation rendering library: Render3DModel

[0736] Overview of data processing and calculation

[0737] 1. Acquiring video data

[0738] The server uses a camera to record the sign language actions performed by the user. The camera in the smart glasses captures the sign language in real time and sends the video data to the server.

[0739] 2. Analysis of video data

[0740] The acquired video data is split into frames on the server, and computer vision algorithms such as OpenCV are used to analyze the shape and position of the hand in each frame.

[0741] 3. Sign Language Identification

[0742] The analysis results of hand shape and position are input into a sign language recognition model (HandSignRecognitionModel), which uses a deep learning algorithm to analyze the hand movements in each frame and identify the corresponding sign language word or phrase.

[0743] 4. Conversion to Character Data

[0744] The sign language identified by the sign language recognition model is converted into the corresponding text on the server and sent to a display-capable device or smart glasses, which then display the text information on the display in real time.

[0745] 5. Converting character data into sign language motion data

[0746] The server inputs character data from the user into a TextToSignModel and generates corresponding sign language motion data, which is then rendered into a 3D animation format using a Render3DModel.

[0747] 6. Displaying Sign Language Animation

[0748] The generated 3D animation is sent from the server to smart glasses or other display-capable devices and displayed in real time.

[0749] Specific operation example

[0750] Example 1: Sign language to text conversion

[0751] 1. The user signs "hello."

[0752] 2. The camera in the smart glasses captures the sign language and sends the video to a server.

[0753] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[0754] 4. The server uses a sign language recognition model to identify this movement as "hello."

[0755] 5. The server generates the string "hello" and sends it to the smart glasses.

[0756] 6. The smart glasses will display this string in real time.

[0757] Example 2: Converting text to sign language

[0758] 1. The user types "thank you" into their smartphone.

[0759] 2. The entered character data is sent to the server.

[0760] 3. The server uses TextToSignModel to generate sign language motion data corresponding to "thank you."

[0761] 4. The server renders this motion into a 3D animation and sends it to the smart glasses.

[0762] 5. The smart glasses play the animation and display it to the user.

[0763] Prompt Sentence Examples

[0764] Examples of prompts for generative AI models include:

[0765] prompt:

[0766] A user signs "Hello." This video is captured by a camera and sent to a sign language recognition system. The system uses a sign language recognition model to analyze the movement and generate the corresponding text "Hello." This text should then be displayed on the smart glasses.

[0767] output:

[0768] The text "Hello" will appear on the smartglasses display.

[0769] This will enable real-time conversion between sign language and text using smart glasses during face-to-face communication in brick-and-mortar stores, enabling smooth communication with the hearing impaired.

[0770] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0771] Step 1:

[0772] The user signs.

[0773] Input: The user's sign language action.

[0774] Output: Video data captured by the camera.

[0775] Specific operation: The camera on the device (smart glasses) captures the user's sign language movements in real time and acquires them as video data.

[0776] Step 2:

[0777] The video data acquired by the terminal is sent to the server.

[0778] Input: Video data captured by the camera.

[0779] Output: Video data sent to the server.

[0780] Specific operation: The network connection is used to send the video data acquired by the device to the server in real time.

[0781] Step 3:

[0782] The server divides the video data into frames and analyzes the shape and position of the hand in each frame.

[0783] Input: Video data received by the server.

[0784] Output: Analysis results of hand shape and position for each frame.

[0785] How it works: The server divides the video data into frames and uses computer vision algorithms such as OpenCV to identify the shape and position of the hand for each frame.

[0786] Step 4:

[0787] The server inputs the analysis results into a sign language recognition model (HandSignRecognitionModel) and generates a corresponding string.

[0788] Input: Analysis results of hand shape and position.

[0789] Output: The corresponding string data.

[0790] How it works: The server inputs hand shape and position data into a sign language recognition model, then uses a deep learning algorithm to analyze the sign language and generate a corresponding string of characters.

[0791] Step 5:

[0792] The server sends the generated string to a display-capable device or smart glasses.

[0793] Input: String data generated by a sign language recognition model.

[0794] Output: The string to be displayed on the smart glasses.

[0795] Specific operation: The server sends the generated string data to the smart glasses via the network and displays it in real time.

[0796] Step 6:

[0797] The user inputs character data.

[0798] Input: Character data entered by the user.

[0799] Output: Character data entered into the terminal.

[0800] Specific operation: The user inputs text data (e.g., "Thank you") using a device such as a smartphone or tablet.

[0801] Step 7:

[0802] The terminal transmits the input character data to the server.

[0803] Input: Character data entered by the user into a terminal.

[0804] Output: Character data sent to the server.

[0805] Specific operation: The terminal transmits the input character data to the server in real time via the network.

[0806] Step 8:

[0807] The server converts the character data into sign language motion data.

[0808] Input: Character data received by the server.

[0809] Output: The corresponding sign language motion data.

[0810] Specific operation: The server uses TextToSignModel to generate sign language motion data from character data.

[0811] Step 9:

[0812] The server renders the sign language motion data into a 3D animation format.

[0813] Input: Generated sign language motion data.

[0814] Output: Sign language motion in 3D animation format.

[0815] Specific operation: The server uses Render3DModel to convert sign language motion data into 3D animation.

[0816] Step 10:

[0817] The server sends the rendered 3D animation to a device capable of displaying it.

[0818] Input: Sign language motion in 3D animation format.

[0819] Output: Sign language animation displayed on smart glasses.

[0820] Specific operation: The server sends the generated 3D animation to the smart glasses via the network and displays it in real time.

[0821] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0822] This invention adds an emotion engine that recognizes the user's emotions to systems that translate sign language into text and systems that translate text into sign language, thereby improving the accuracy of sign language translation and realizing natural communication that takes emotions into account.

[0823] Translating sign language into text

[0824] 1. Acquiring video data

[0825] When the user signs, the device uses its camera to capture video of the sign.

[0826] The video data captured by the device is sent to the server in real time.

[0827] 2. Analysis of video data

[0828] The video data received by the server is divided into frames, and the position and shape of the hand in each frame is analyzed using a computer vision algorithm.

[0829] The server extracts the user's facial expressions from the video frames and recognizes the user's emotions using an emotion engine that uses facial recognition algorithms and natural language processing algorithms.

[0830] 3. Sign Language Identification

[0831] The server inputs the analysis results into a sign language recognition model (e.g., a deep learning model), which analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[0832] The system adjusts responses by combining recognized sign language with emotions inferred from the user's facial expressions.

[0833] 4. Conversion to Character Data

[0834] The server converts sign language into corresponding text, selecting an appropriate expression that reflects the user's emotions.

[0835] The server sends the converted string to the terminal and displays it to the user.

[0836] Example (translating sign language into text)

[0837] 1. The user smiles and signs "hello."

[0838] 2. The device uses its camera to capture sign language and facial expressions and sends the video to the server.

[0839] 3. The server divides the video into frames and analyzes the shape and position of the hands and facial expressions.

[0840] 4. The server uses the sign language recognition model to identify "hello" and the emotion engine recognizes a smile.

[0841] 5. The server generates the string "hello" and displays it in a friendly way.

[0842] 6. The terminal displays this string to the user.

[0843] Translating text into sign language

[0844] 1. Entering text data

[0845] The user inputs text data (e.g., "Thank you") into the terminal.

[0846] The terminal transmits the input character data to the server.

[0847] 2. Sign Language Generation

[0848] The server inputs the character data into a sign language generation algorithm to generate corresponding sign language motion data.

[0849] The server uses an emotion engine to predict facial expressions from input text and adjusts motion data based on that information.

[0850] 3. Sending sign language animations

[0851] The server renders the generated sign language motion data into a 3D animation format.

[0852] The server sends the rendered sign language animation data to the terminal.

[0853] 4. Displaying Sign Language Animation

[0854] The terminal plays back the received animation to the user and displays it as sign language motion that includes emotion.

[0855] Example (translating text into sign language)

[0856] 1. The user types the word "thank you" into the terminal.

[0857] 2. The terminal sends the character data to the server.

[0858] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[0859] 4. The server uses an emotion engine to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[0860] 5. The server renders this motion into a 3D animation and sends it back to the device.

[0861] 6. The device plays the animation and displays the sign language animation that reflects the emotion to the user.

[0862] This allows users to communicate naturally in real time, and conversion between sign language and text is performed in a way that includes emotional information.

[0863] The processing flow will be explained below.

[0864] Translating sign language into text

[0865] Step 1:

[0866] The user signs, using natural facial expressions and eye contact along with hand movements.

[0867] Step 2:

[0868] The device uses a camera to capture video of the user's sign language and facial expressions, and sends the captured video data to a server in real time.

[0869] Step 3:

[0870] The video data received by the server is divided into frames, which is a process of converting the video file into still image frames.

[0871] Step 4:

[0872] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts features of the hand shape, position, and movement.

[0873] Step 5:

[0874] The server analyzes the user's face in the video frames and uses facial recognition algorithms to recognize facial expressions, as well as natural language processing algorithms to infer emotions from facial expressions.

[0875] Step 6:

[0876] The server feeds the analyzed frames into a sign language recognition model (e.g., a deep learning model), which identifies the sign language words or phrases that correspond to the movements in each frame.

[0877] Step 7:

[0878] The server converts the sign language into the corresponding text based on the output of the sign language recognition model. At this time, the server adjusts the expression of the text taking into account the user's emotion estimated by the emotion engine.

[0879] Step 8:

[0880] The server sends the converted string data to the terminal.

[0881] Step 9:

[0882] The terminal displays the character string data received from the server to the user in an appropriate format that reflects the user's emotions.

[0883] Translating text into sign language

[0884] Step 1:

[0885] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[0886] Step 2:

[0887] The terminal sends the input or converted character data to the server.

[0888] Step 3:

[0889] The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data based on the text data.

[0890] Step 4:

[0891] The server uses an emotion engine to recognize the emotional tone from the input text data and voice data, and adjusts the sign language motion data based on the emotion estimated from the text data and voice data.

[0892] Step 5:

[0893] The server renders 3D animation based on the generated sign language motion data, which visualizes natural sign language motion that reflects emotional information.

[0894] Step 6:

[0895] The server sends the rendered sign language animation data to the terminal.

[0896] Step 7:

[0897] The terminal then plays back the received sign language animation and displays it to the user. This display is a natural sign language animation that reflects the user's emotions.

[0898] This allows users to easily translate between sign language and text in real time, incorporating emotions.

[0899] Example 2

[0900] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0901] When communicating between sign language users and other people, there is a need for a system that can accurately understand sign language and express emotions in real time. However, existing sign language translation systems have low sign language recognition accuracy and do not take into account the user's emotional expression, making natural communication difficult. To solve this problem, a system is needed that improves the accuracy of sign language recognition and translates the user's emotions.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0903] In this invention, the server includes means for capturing a video of a user's sign language movements, means for dividing the captured video into frames and analyzing the hand shape and position of each frame, means for extracting the user's facial expressions from the captured video frames and recognizing their emotions, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, and means for displaying the generated character string to the user. This enables highly accurate recognition of the user's sign language and natural translation that reflects the user's emotions.

[0904] "Means for capturing video of the actions of a sign language user" refers to devices such as cameras and sensors for recording the sign language actions of a sign language user, and includes technology for collecting those actions as digital video data.

[0905] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a software algorithm that breaks down the recorded video data into individual frames and identifies the position and shape of the user's hand in each frame. Specifically, it utilizes computer vision technology.

[0906] "Means for extracting a user's facial expressions from captured video frames and recognizing emotions" refers to a facial recognition algorithm and emotion analysis engine that analyzes facial features contained in video frames and grasps the user's emotional state.

[0907] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to the process of inputting the analyzed hand movement and shape data into a sign language recognition model (e.g., a model using a deep learning algorithm) and outputting the corresponding string of characters as a result.

[0908] "Means for displaying the generated character string to the user" refers to a device or software component for visually displaying the character string sent from the server to the user. Specifically, this includes the display of a smartphone or a dedicated display device.

[0909] "Means for converting character data into sign language motion data" refers to a software algorithm for converting input character string data into corresponding sign language motion data. Specifically, it includes a sign language dictionary database and a motion generation model.

[0910] "Means for rendering sign language motion data into a 3D animation format" refers to technology for rendering sign language motion data as 3D computer graphics to generate visually realistic sign language animation. Specifically, this refers to the use of 3D modeling software and rendering engines.

[0911] "Means for displaying the rendered animation to the user" refers to a video player component or display device that plays the generated 3D animation on the user's device, including a computer or mobile device display.

[0912] The present invention adds an emotion engine that recognizes the user's emotions to a system that translates sign language into text and a system that translates text into sign language. This improves the accuracy of sign language translation and realizes natural communication that reflects emotions. Specific embodiments of the present invention are described below.

[0913] Translating sign language into text

[0914] 1. Acquiring video data

[0915] The user signs.

[0916] The device (smartphone or camera-equipped device) records the sign language as a video, capturing the video in real time using the device's built-in camera.

[0917] The video data captured by the device is encoded in real time and sent to a server via the Internet using the RTMP protocol or WebRTC.

[0918] 2. Analysis of video data

[0919] The video data received by the server is divided into frames. This process uses a library such as FFmpeg.

[0920] The server analyzes each frame and uses OpenCV to identify the position and shape of the hand, using algorithms such as Edge Detection and Contour Detection.

[0921] The server uses a facial recognition algorithm (dlib, FaceNet, etc.) to extract the user's facial expressions from video frames, and then uses an emotion engine that utilizes natural language processing algorithms such as BERT to recognize the user's emotions.

[0922] 3. Sign Language Identification

[0923] The server inputs the analysis results into a sign language recognition model (a deep learning model built with TensorFlow or PyTorch), which analyzes the hand movements in each frame and identifies the corresponding sign language words and phrases.

[0924] The server combines the sign language recognition results with the emotions inferred by the emotion engine and adjusts the response.

[0925] 4. Conversion to Character Data

[0926] The server converts the sign language into a corresponding string of characters, choosing an appropriate expression that reflects the user's emotions. For example, for the sign "hello," the server converts it into "hello!", taking into account the smile.

[0927] The server encodes the generated character data and sends it to the terminal via the Internet.

[0928] The terminal decodes the received character data and displays it on the user's screen. Specifically, this is the operation of setting the character string in a GUI component.

[0929] Example (translating sign language into text)

[0930] The user smiles and signs "hello."

[0931] The device uses a camera to capture the sign language and facial expressions, and then sends the video to a server via the RTMP protocol.

[0932] The server splits the video into frames, analyzes hand shape and position using OpenCV, and analyzes facial expressions using dlib.

[0933] The server identifies "hello" using a TensorFlow model, and the emotion engine recognizes a smile using BERT.

[0934] The server generates the string "Hello!" and sends it to the terminal.

[0935] The terminal displays this string to the user.

[0936] Translating text into sign language

[0937] 1. Entering text data

[0938] The user enters text data (e.g., "Thank you") into the terminal, using a touchscreen or keyboard.

[0939] The device sends the input character data to the server via HTTP or WebSocket.

[0940] 2. Sign Language Generation

[0941] The server receives the character data and inputs it into a sign language generation algorithm (e.g., a deep learning model), which searches for corresponding sign language motion data from a character-to-sign language dictionary database.

[0942] The server uses an emotion engine to predict facial expressions from the input text and adjusts the sign language motion data based on that information. Specifically, it adjusts the movement and speed of the motion data to correspond to the emotion.

[0943] 3. Sending sign language animations

[0944] The server renders the generated sign language motion data using Blender or other 3D rendering software, and animates the 3D model of the sign language character.

[0945] The server encodes the 3D animation data and sends it to the device via the Internet.

[0946] 4. Displaying Sign Language Animation

[0947] The device decodes the received animation and plays it on the user's screen. Specifically, this is done by playing it using the video player component in the app.

[0948] The device displays emotional sign language motions to the user in 3D animation format.

[0949] Example (translating text into sign language)

[0950] The user types the word "thank you" into the device, for example, using the keyboard or touch screen of a smartphone.

[0951] The terminal sends the character data to the server via HTTP.

[0952] The server uses a sign language generation algorithm (TensorFlow model) to generate sign language motion data corresponding to "thank you."

[0953] The server uses an emotion engine (using the BERT model) to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[0954] The server uses Blender to render this motion data into a 3D animation, encodes it, and then sends it to the device.

[0955] The device decodes the animation and uses a video player to play the emotion-reflecting sign language animation on the user's screen.

[0956] Example prompts to input to the generative AI model

[0957] 1. Translating sign language to text: "When a user says 'hello' in sign language, analyze their hand movements and facial expressions and generate an appropriate string of text based on their emotions."

[0958] 2. Example of translating text into sign language: "Generate motion data to express the word 'thank you' in sign language, and create a 3D animation that reflects the emotion of gratitude."

[0959] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0960] Translating sign language into text

[0961] Step 1:

[0962] The user signs

[0963] Input: User's sign language actions

[0964] Specific action: The user signs to the camera, for example, signing "hello."

[0965] Output: Video data including the user's sign language

[0966] Step 2:

[0967] The device captures video and sends it to the server in real time.

[0968] Input: Video data containing the user's sign language

[0969] Specific operation: The system uses the device's camera to capture sign language video in real time, encodes the video data using the RTMP protocol, and then transmits the encoded video data to a server via the Internet.

[0970] Output: Video data sent to the server

[0971] Step 3:

[0972] The server divides the video data into frames and analyzes the shape and position of the hand.

[0973] Input: Video data

[0974] How it works: The server splits the video data into frames using the FFmpeg library. For each frame, it analyzes the hand shape and position using the Edge Detection and Contour Detection algorithms using OpenCV.

[0975] Output: Analysis results of hand shape and position

[0976] Step 4:

[0977] The server extracts facial expressions from video frames and recognizes emotions.

[0978] Input: Video data

[0979] Specific operation: The server uses dlib and FaceNet to detect facial features from frames, and uses BERT, a natural language processing algorithm, to recognize the user's emotions from the detected facial features.

[0980] Output: User emotion data

[0981] Step 5:

[0982] The server inputs the analysis results into a sign language recognition model and generates the corresponding string.

[0983] Input: Analysis results of hand shape and position, user emotion data

[0984] How it works: The server inputs the analysis results into a deep learning model built with TensorFlow or PyTorch, converts the sign language gestures into corresponding text, and uses the emotion data to select the appropriate expression method as needed.

[0985] Output: The corresponding string

[0986] Step 6:

[0987] The server sends the generated string to the terminal, which displays the string to the user.

[0988] Input: The corresponding string

[0989] Specific operation: The server encodes the generated string and sends it to the terminal via the Internet. The terminal decodes the received string and displays it in the GUI component.

[0990] Output: The string displayed on the user's screen

[0991] Translating text into sign language

[0992] Step 1:

[0993] The user enters character data

[0994] Input: Text data (e.g. "Thank you")

[0995] Specific Action: A user types text using the keyboard or touchscreen of a smartphone or tablet.

[0996] Output: Character data entered

[0997] Step 2:

[0998] The terminal sends the input character data to the server.

[0999] Input: Character data entered

[1000] Specific operation: The terminal sends character data to the server using HTTP or WebSocket.

[1001] Output: Character data sent to the server

[1002] Step 3:

[1003] The server converts the text data into sign language motion data.

[1004] Input: Character data

[1005] How it works: The server inputs character data into a sign language generation algorithm (e.g., a TensorFlow model) to generate sign language motion data. It then uses an emotion engine to infer emotions from the character data and adjusts the motion data based on the results.

[1006] Output: Sign language motion data

[1007] Step 4:

[1008] The server renders the sign language motion data into a 3D animation format.

[1009] Input: Sign language motion data

[1010] Specific operation: The server uses 3D rendering software such as Blender to render the sign language motion data into 3D animation.

[1011] Output: 3D animation data

[1012] Step 5:

[1013] The server sends 3D animation data to the terminal, and the terminal displays the animation to the user.

[1014] Input: 3D animation data

[1015] Specific operation: The server encodes 3D animation data and sends it to the device via the Internet. The device decodes the received animation data and plays it using the video player component.

[1016] Output: The animation played on the user's screen

[1017] (Application example 2)

[1018] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1019] Conventional sign language translation systems simply convert sign language into text, making it difficult to achieve natural communication that reflects the user's emotions. Furthermore, they lacked the ability to output text generated from sign language as speech, or the ability to convert text into sign language animation. This made it difficult for hearing-impaired people and people who use sign language to communicate smoothly on public transportation, such as self-driving vehicles.

[1020] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a video of a user's actions using sign language, means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, means for extracting the user's facial expression from the video data and recognizing the emotion using an emotion engine, means for adjusting the character string to reflect the user's emotion, means for outputting the generated character string as audio, means for converting character data into sign language motion data, means for rendering the sign language motion data in a 3D animation format, and means for reflecting emotion in the rendered animation using the emotion engine. This enables users to communicate naturally and emotionally even inside an autonomous vehicle.

[1021] 1. "Means for capturing the actions of a user using sign language as video" refers to a device or system that records the actions of a user using sign language in video format using an image capture device such as a camera.

[1022] 2. "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to an algorithm or software that breaks down video data into individual frames and analyzes the shape and position of the user's hand within each frame.

[1023] 3. "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to a process or system that inputs the analyzed hand movement data into a sign language recognition model and generates a corresponding string of characters as a result.

[1024] 4. "Means for displaying the generated character string to the user" refers to a device or system that outputs the generated character data to a display or other display device and presents it in a form that the user can visually confirm.

[1025] 5. "Means for extracting a user's facial expressions from video data and recognizing emotions using an emotion engine" means a process or system that extracts a user's facial expressions from video data and identifies the user's emotional state using machine learning or artificial intelligence algorithms.

[1026] 6. "Means for adjusting text to reflect user emotions" refers to an algorithm or process that appropriately changes the content and expression of generated text based on recognized user emotional information.

[1027] 7. "Means for audibly outputting the generated character string" refers to a device or system that uses speech synthesis technology to play back the generated character data as speech.

[1028] 8. "Means for converting character data into sign language motion data" means an algorithm or software that converts input character information into corresponding sign language motion data.

[1029] 9. "Means for rendering sign language motion data into a 3D animation format" means a process or system that renders the generated sign language motion data as a three-dimensional animation.

[1030] 10. "Means for reflecting emotions in rendered animation using an emotion engine" means a system or algorithm that applies an emotion engine to generated 3D animation and reflects emotions in the character's movements and expressions.

[1031] This invention relates to a system that enables users who use sign language to communicate naturally and emotionally within an autonomous vehicle. This system captures sign language movements as video and generates text, audio, and sign language animations based on the analysis results, thereby supporting comfortable communication for users.

[1032] System Program

[1033] The system uses the following hardware and software:

[1034] Hardware: In-vehicle camera, high-resolution display, on-board computer, audio output device

[1035] Software: Computer vision libraries (e.g., OpenCV), deep learning frameworks (e.g., TensorFlow, PyTorch), natural language processing libraries (e.g., NLTK, Transformers), emotion recognition engines

[1036] System action

[1037] 1. Acquiring video data:

[1038] The in-vehicle camera captures the user's sign language in real time as video, which is then broken down into frames and the shape and position of the hand in each frame are analyzed.

[1039] 2. Sign Language Analysis and Recognition:

[1040] The analyzed data is input into a sign language recognition model (deep learning algorithm) to generate a corresponding string of characters. At the same time, the user's facial expressions are extracted from the video data, and an emotion engine recognizes the user's emotions.

[1041] 3. Creating and displaying strings:

[1042] The system adjusts the text based on the recognized sign language and emotional information, and the resulting text is displayed on the vehicle's display. The text is also output as speech using speech synthesis technology.

[1043] 4. Generating Sign Language Animation:

[1044] Conversely, text data entered by the user is converted into sign language motion data, which is then rendered into a 3D animation format and emotion-based using an emotion recognition engine. The final rendered animation is then displayed on the screen.

[1045] Specific examples

[1046] Translating sign language into text and speech

[1047] 1. A user signs "thank you" inside a self-driving vehicle.

[1048] 2. Cameras inside the vehicle capture sign language in real time and transmit the data to an onboard computer.

[1049] 3. The on-board computer divides the video into frames and analyzes hand movements and the user's facial expressions.

[1050] 4. Based on the analysis results, the sign language recognition model and emotion engine generate the string "Thank you" and emotional information.

[1051] 5. "Thank you" will appear on the display and be output as audio at the same time.

[1052] Translating text into sign language animation

[1053] 1. The user types "Help me" into the display.

[1054] 2. The on-board computer converts the text data into sign language motion data, and the emotion engine recognizes and reflects the emotion of "urgency."

[1055] 3. The generated sign language motion data is rendered as a 3D animation and displayed on the screen.

[1056] Prompt Sentence Examples

[1057] Translate sign language to text: A user signs "Where is my house?" You can see confusion in the user's face.

[1058] Information from sign language:

[1059] Transcription: Where is my house?

[1060] Emotion: Confused

[1061] Translate text into sign language: The user has typed "Thank you." Reflect the sentiment of gratitude with the emotion engine.

[1062] Generated sign language motion:

[1063] Sign Language: Thank you

[1064] Emotion: Gratitude

[1065] This will enable users to have natural and emotionally rich communication even inside an autonomous vehicle.

[1066] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1067] Step 1:

[1068] Acquiring video data using a camera: When a user signs, the device's in-car camera captures the action in real time. This captures video data including the user's hand movements and facial expressions. The input is "video data including the user's sign language and facial expressions," and the output is "video data acquired in real time."

[1069] Step 2:

[1070] Video data frame division and analysis: The server receives the acquired video data and divides the video into frames. The shape and position of the hand in each frame are analyzed using a computer vision library (e.g., OpenCV). The input is "video data acquired in real time," and the output is "hand shape and position data for each frame."

[1071] Step 3:

[1072] Sign language recognition: The server inputs the analyzed frame data into a sign language recognition model (e.g., a deep learning algorithm using TensorFlow or PyTorch) to identify the corresponding sign language words or phrases. The input is the hand shape and position data for each frame, and the output is the identified sign language string.

[1073] Step 4:

[1074] Facial expression analysis and emotion recognition: The server extracts the user's facial expressions from the video data and recognizes the user's emotions using a natural language processing algorithm and an emotion engine. The input is "video data acquired in real time" and the output is "user's emotional information."

[1075] Step 5:

[1076] Generating and displaying text and audio: The server adjusts the text based on the recognized sign language text and emotional information, and displays it on the vehicle's display. The text is also output as audio using a speech synthesis engine. The input is the "identified sign language text and the user's emotional information," and the output is "text and audio data displayed to the user."

[1077] Step 6:

[1078] Conversion of character data into sign language motion data: The server receives character data entered by the user and converts it into corresponding sign language motion data. The input is "character data entered by the user" and the output is "sign language motion data."

[1079] Step 7:

[1080] Rendering sign language motion data and reflecting emotions: The server renders the sign language motion data into 3D animation format and reflects emotional information using an emotion engine. The input is "sign language motion data" and "emotional information," and the output is "emotionally reflected sign language 3D animation."

[1081] Step 8:

[1082] Displaying sign language animation: The device displays the rendered sign language 3D animation on the display, providing a visual experience to the user. The input is "emotion-reflecting sign language 3D animation," and the output is "sign language 3D animation displayed to the user."

[1083] Through these steps, users can communicate naturally and emotively using sign language inside a self-driving vehicle.

[1084] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1085] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1086] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1087] [Third embodiment]

[1088] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1089] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1090] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1091] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1092] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1093] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1094] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1095] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1096] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1097] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1098] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1099] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1100] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[1101] Translating sign language into text

[1102] 1. Acquiring video data

[1103] When the user signs, the device uses its camera to capture video of the sign.

[1104] The captured video data is sent to a server in real time.

[1105] 2. Analysis of video data

[1106] The video received by the server is divided into frames.

[1107] The hand position and shape in each frame are analyzed using a computer vision algorithm (e.g., OpenCV).

[1108] 3. Sign Language Identification

[1109] The server inputs the analyzed data into a sign language recognition model (e.g., a deep learning model).

[1110] The sign language recognition model analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[1111] 4. Conversion to Character Data

[1112] The server converts the identified sign language into a corresponding string.

[1113] The converted string is sent to the terminal, which displays the characters to the user.

[1114] Example (translating sign language into text)

[1115] 1. The user signs "hello."

[1116] 2. The device captures the sign language with its camera and sends the video to the server.

[1117] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[1118] 4. The server uses the recognition model to determine that this motion is a "hello."

[1119] 5. The server generates the string "Hello" and sends it to the terminal.

[1120] 6. The terminal displays this string to the user.

[1121] Translating text into sign language

[1122] 1. Entering text data

[1123] The user inputs text data (for example, "Thank you") into the terminal.

[1124] In the case of a live voice, the audio is recorded using the device's microphone and the audio data is converted into text.

[1125] The text data is sent to the server.

[1126] 2. Sign Language Generation

[1127] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data.

[1128] Once the motion data is generated, the server renders it into a 3D animation format.

[1129] 3. Sending sign language animations

[1130] The server transmits the generated sign language animation data to the terminal.

[1131] Users can view sign language animations on their devices.

[1132] Example (translating text into sign language)

[1133] 1. The user types the word "thank you" into the terminal.

[1134] 2. The terminal sends the character data to the server.

[1135] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[1136] 4. The server renders this motion into a 3D animation and sends it back to the device.

[1137] 5. The device plays the animation and displays it to the user.

[1138] This allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motion, enabling smooth communication with the hearing impaired.

[1139] The processing flow will be explained below.

[1140] Translating sign language into text

[1141] Step 1:

[1142] The user signs a particular phrase or word.

[1143] Step 2:

[1144] The device uses a camera to record the user's sign language as a video, and the video data is sent to a server in real time.

[1145] Step 3:

[1146] The video data received by the server is divided into frames. This division is the process of converting the video into individual still images (frames).

[1147] Step 4:

[1148] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts data on the hand's shape, position, and movement characteristics.

[1149] Step 5:

[1150] The server feeds the parsed frames into a sign language recognition model (e.g., a deep learning model), which identifies sign language words and phrases based on hand movements and position information.

[1151] Step 6:

[1152] The server converts the signs that are identified with high confidence into corresponding strings of characters based on the output from the sign language recognition model.

[1153] Step 7:

[1154] The server sends the converted string data to the terminal.

[1155] Step 8:

[1156] The terminal displays the character string data received from the server to the user.

[1157] Translating text into sign language

[1158] Step 1:

[1159] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[1160] Step 2:

[1161] The terminal transmits the input character data to the server.

[1162] Step 3:

[1163] The server inputs the text data into a sign language generation algorithm, which converts the text data into sign language motion data.

[1164] Step 4:

[1165] The server renders 3D animation based on the generated sign language motion data. During this rendering process, the sign language motion is visually represented.

[1166] Step 5:

[1167] The server sends the rendered sign language animation data to the terminal.

[1168] Step 6:

[1169] The terminal plays the received sign language animation and displays it to the user.

[1170] This allows users to input sign language and convert it into text, and input text or voice and convert it into sign language in real time.

[1171] Example 1

[1172] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] Conventional sign language translation systems have had difficulty accurately recognizing sign language movements and converting them into text in real time. Furthermore, when converting text data into sign language motion, there was a lack of technology to generate natural-looking sign language animations. As a result, smooth communication with the hearing impaired was difficult.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1175] In this invention, the server includes means for capturing user actions as video, means for dividing the captured video into frames and analyzing the hand shape and position in each frame, means for inputting the analysis results into a sign language recognition model and converting the sign language actions into corresponding character strings, and means for displaying the generated character strings to the user in real time, thereby enabling the sign language actions to be accurately and quickly converted into character strings and displayed to the user.

[1176] In this invention, the server further includes means for converting character data into sign language motion data, means for rendering the sign language motion data into a 3D animation format, and means for displaying the rendered animation to the user, thereby enabling input character data to be displayed as natural sign language animation.

[1177] "User" refers to a person who uses sign language or inputs character data.

[1178] "Terminal" refers to a device that includes a camera that captures video of the user signing and a device that inputs text data.

[1179] "Server" refers to a computer system that receives and analyzes video and text data sent from a terminal, and performs sign language recognition and generates 3D animation.

[1180] "Video data" refers to video information captured by a camera of a user's sign language actions.

[1181] A "frame" refers to each still image that makes up video data.

[1182] "Computer vision algorithm" refers to image processing technology for analyzing the position and shape of hands from video data.

[1183] "Sign language recognition model" refers to a deep learning model for identifying sign language words and phrases based on analyzed hand movements.

[1184] "Character string" refers to text information in which sign language actions are converted into corresponding characters.

[1185] "Sign language motion data" refers to data that expresses sign language movements and is generated based on character data.

[1186] "3D animation format" refers to a format that expresses sign language motion data as three-dimensional movement.

[1187] "Real-time" means that the system processes and converts data instantly and displays the results without delay.

[1188] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[1189] Translating sign language into text

[1190] When a user signs, the device uses a camera to capture video of the sign language. Specifically, a camera installed on a smartphone or PC is used. The captured video data is sent to a server in real time. The server divides the received video into frames and analyzes the position and shape of the hands in each frame. This analysis is performed using computer vision algorithms such as OpenCV. The analyzed data is input into a sign language recognition model. A deep learning model built using TensorFlow or PyTorch is used as the sign language recognition model. The recognition model analyzes the hand movements in each frame and identifies the corresponding sign language word or phrase. The identified sign language is converted into the corresponding string of characters, and the converted string is sent to the device. The device then displays the characters to the user.

[1191] For example, if a user signs "hello," the device captures the sign with a camera and sends the video to the server. The server divides the video into frames and recognizes the shape and position of the hand. The server then uses a recognition model to determine that this movement is "hello." The server generates the string "hello" and sends it to the device. The device then displays this string to the user.

[1192] Translating text into sign language

[1193] The user inputs text data (for example, "Thank you") into the device. If the input is human voice, the audio is recorded using the device's microphone, and the audio data is converted into text using a method such as the Google Cloud Speech-to-Text API. This text data is sent to the server. The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data. The generated motion data is rendered in a 3D animation format. Rendering is done using Unity or Blender. The generated sign language animation data is sent to the device, and the user can view the sign language animation on the device.

[1194] For example, if a user types the character "Thank you" into a device, the device sends the character data to the server. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "Thank you." The server renders this motion into a 3D animation and sends it back to the device. The device plays the animation and displays it to the user.

[1195] This system allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motions, enabling smooth communication with the hearing impaired.

[1196] Prompt Sentence Examples

[1197] "Please explain how this system translates sign language into text in real time and simultaneously renders sign language animations in multiple dimensions."

[1198] Hardware and software used

[1199] Hardware: smartphone, computer, camera, microphone

[1200] Software: OpenCV, TensorFlow, PyTorch, Google Cloud Speech-to-Text API, Unity, Blender

[1201] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1202] Step 1:

[1203] Acquiring video data

[1204] Input: User's sign language actions

[1205] The device captures the user's sign language movements with a camera, which can be a built-in camera on a smartphone or PC, and transmits the captured video data to a server in real time.

[1206] Step 2:

[1207] Video data analysis

[1208] Input: Video data from the device

[1209] Output: Hand position and shape data per frame

[1210] The server splits the received video into frames using a library such as FFmpeg. The position and shape of the hand in each frame is analyzed using a computer vision algorithm (e.g., OpenCV). For example, the position of the hand can be identified by detecting a specific color area within the frame.

[1211] Step 3:

[1212] Sign Language Identification

[1213] Input: Parsed frame data

[1214] Output: A string corresponding to the sign language action

[1215] The server inputs the parsed data into a sign language recognition model, which uses deep learning models built with TensorFlow and PyTorch. The model analyzes the hand movements in each frame and identifies which sign language words or phrases they correspond to. For example, a series of hand movements may have a pattern that corresponds to a specific sign language word.

[1216] Step 4:

[1217] Conversion to character data

[1218] Input: Recognition results of sign language recognition model

[1219] Output: The corresponding string

[1220] The server converts the sign language actions identified by the sign language recognition model into a corresponding string. For example, if the hand movement is identified as corresponding to "hello," the server generates the string "hello." The server then transmits the generated string to the terminal.

[1221] Step 5:

[1222] Displaying a string

[1223] Input: The string sent from the server

[1224] Output: The string the user sees on the terminal

[1225] The terminal displays the text sent from the server on the screen in real time using a standard text drawing library.

[1226] Step 6:

[1227] Entering character data

[1228] Input: Character data entered by the user.

[1229] The user inputs text data (for example, "Thank you") into the device. In the case of voice input, the voice is recorded with the device's microphone and converted into text using the Google Cloud Speech-to-Text API or similar. The text data is then sent to the server.

[1230] Step 7:

[1231] Generating sign language motion data

[1232] Input: Character data

[1233] Output: Sign language motion data

[1234] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data, which expresses the sign language movements in detail.

[1235] Step 8:

[1236] 3D animation rendering

[1237] Input: Sign language motion data

[1238] Output: 3D animation data

[1239] The server renders the sign language motion data into a 3D animation format using Unity or Blender to visually represent the motion data.

[1240] Step 9:

[1241] Playing and viewing animations

[1242] Input: 3D animation data

[1243] Output: Sign language animation that the user sees on their device

[1244] The server sends the generated 3D animation data to the device, which then plays the animation and displays it to the user, allowing the user to see the sign language through the animation.

[1245] (Application example 1)

[1246] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1247] Conventional sign language recognition systems can convert sign language into text and vice versa, but they lack the support functions to facilitate smooth face-to-face communication in brick-and-mortar stores, etc. In particular, there is a demand for a system that uses smart glasses to convert between sign language and text in real time and display sign language motion as 3D animation, enabling smooth communication with the hearing impaired.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1249] In this invention, the server includes: means for capturing a video of a user's sign language movements; means for dividing the captured video into frames and analyzing the hand shape and position of each frame; means for inputting the analysis results into a sign language recognition model and generating a corresponding character string; means for displaying the generated character string on a displayable device; means for displaying the generated character string on smart glasses; means for converting character data into sign language motion data; means for rendering the sign language motion data into a 3D animation format; and means for displaying the rendered animation on a displayable device. This enables real-time conversion between sign language and characters using smart glasses in face-to-face communication such as in brick-and-mortar stores, enabling smooth communication with hearing-impaired people.

[1250] "Means for capturing the actions of a user performing sign language as video" refers to a device that uses a camera or other sensor to record the sign language movements of a user as video data.

[1251] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a system that divides the recorded video into frames and uses algorithms and computer vision technology to analyze the position and shape of the hand in each frame.

[1252] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding character string" refers to the process of inputting the analysis results of hand position and shape into a sign language recognition model and generating a corresponding character string as a result. This sign language recognition model utilizes technologies such as machine learning and deep learning.

[1253] The "means for presenting the generated character string on a displayable device" refers to a method for outputting the generated character string to a display device such as a smartphone, tablet, or computer display.

[1254] The "means for displaying the generated character string on the smart glasses" refers to a device that displays the generated character string on the display of the smart glasses in real time. When a user wears the smart glasses, they can simultaneously view the real world and the display.

[1255] "Means for converting character data into sign language motion data" refers to an algorithm or computer program for converting input character data into sign language motion, thereby converting the character information into dynamic sign language expressions.

[1256] "Means for rendering sign language motion data into a 3D animation format" refers to a process or system that generates 3D animation from sign language motion data. This rendering creates a visually realistic 3D animation of the sign language motion.

[1257] A "means for presenting rendered animation on a display-capable device" refers to a method or apparatus for showing the generated 3D animation to a user on a smartphone, tablet, computer display, etc.

[1258] The present invention relates to a system for converting sign language into text in real time, and a system for converting text into sign language. This system has the function of acquiring and analyzing a user's sign language movements and converting them into text, and conversely, converting the text into sign language motion data and displaying it. Specific embodiments of the present invention will be described below.

[1259] Hardware and Software Configuration

[1260] This system is implemented mainly using the following hardware and software.

[1261] Hardware: Smart glasses (with camera and display capabilities), server, displayable devices (smartphones, tablets, computers, etc.)

[1262] software:

[1263] Computer vision algorithms: Image processing libraries such as OpenCV

[1264] Sign language recognition model library: HandSignRecognitionModel

[1265] A library for converting text to sign language motion: TextToSignModel

[1266] 3D animation rendering library: Render3DModel

[1267] Overview of data processing and calculation

[1268] 1. Acquiring video data

[1269] The server uses a camera to record the sign language actions performed by the user. The camera in the smart glasses captures the sign language in real time and sends the video data to the server.

[1270] 2. Analysis of video data

[1271] The acquired video data is split into frames on the server, and computer vision algorithms such as OpenCV are used to analyze the shape and position of the hand in each frame.

[1272] 3. Sign Language Identification

[1273] The analysis results of hand shape and position are input into a sign language recognition model (HandSignRecognitionModel), which uses a deep learning algorithm to analyze the hand movements in each frame and identify the corresponding sign language word or phrase.

[1274] 4. Conversion to Character Data

[1275] The sign language identified by the sign language recognition model is converted into the corresponding text on the server and sent to a display-capable device or smart glasses, which then display the text information on the display in real time.

[1276] 5. Converting character data into sign language motion data

[1277] The server inputs character data from the user into a TextToSignModel and generates corresponding sign language motion data, which is then rendered into a 3D animation format using a Render3DModel.

[1278] 6. Displaying Sign Language Animation

[1279] The generated 3D animation is sent from the server to smart glasses or other display-capable devices and displayed in real time.

[1280] Specific operation example

[1281] Example 1: Sign language to text conversion

[1282] 1. The user signs "hello."

[1283] 2. The camera in the smart glasses captures the sign language and sends the video to a server.

[1284] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[1285] 4. The server uses a sign language recognition model to identify this movement as "hello."

[1286] 5. The server generates the string "hello" and sends it to the smart glasses.

[1287] 6. The smart glasses will display this string in real time.

[1288] Example 2: Converting text to sign language

[1289] 1. The user types "thank you" into their smartphone.

[1290] 2. The entered character data is sent to the server.

[1291] 3. The server uses TextToSignModel to generate sign language motion data corresponding to "thank you."

[1292] 4. The server renders this motion into a 3D animation and sends it to the smart glasses.

[1293] 5. The smart glasses play the animation and display it to the user.

[1294] Prompt Sentence Examples

[1295] Examples of prompts for generative AI models include:

[1296] prompt:

[1297] A user signs "Hello." This video is captured by a camera and sent to a sign language recognition system. The system uses a sign language recognition model to analyze the movement and generate the corresponding text "Hello." This text should then be displayed on the smart glasses.

[1298] output:

[1299] The text "Hello" will appear on the smartglasses display.

[1300] This will enable real-time conversion between sign language and text using smart glasses during face-to-face communication in brick-and-mortar stores, enabling smooth communication with the hearing impaired.

[1301] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1302] Step 1:

[1303] The user signs.

[1304] Input: The user's sign language action.

[1305] Output: Video data captured by the camera.

[1306] Specific operation: The camera on the device (smart glasses) captures the user's sign language movements in real time and acquires them as video data.

[1307] Step 2:

[1308] The video data acquired by the terminal is sent to the server.

[1309] Input: Video data captured by the camera.

[1310] Output: Video data sent to the server.

[1311] Specific operation: The network connection is used to send the video data acquired by the device to the server in real time.

[1312] Step 3:

[1313] The server divides the video data into frames and analyzes the shape and position of the hand in each frame.

[1314] Input: Video data received by the server.

[1315] Output: Analysis results of hand shape and position for each frame.

[1316] How it works: The server divides the video data into frames and uses computer vision algorithms such as OpenCV to identify the shape and position of the hand for each frame.

[1317] Step 4:

[1318] The server inputs the analysis results into a sign language recognition model (HandSignRecognitionModel) and generates a corresponding string.

[1319] Input: Analysis results of hand shape and position.

[1320] Output: The corresponding string data.

[1321] How it works: The server inputs hand shape and position data into a sign language recognition model, then uses a deep learning algorithm to analyze the sign language and generate a corresponding string of characters.

[1322] Step 5:

[1323] The server sends the generated string to a display-capable device or smart glasses.

[1324] Input: String data generated by a sign language recognition model.

[1325] Output: The string to be displayed on the smart glasses.

[1326] Specific operation: The server sends the generated string data to the smart glasses via the network and displays it in real time.

[1327] Step 6:

[1328] The user inputs character data.

[1329] Input: Character data entered by the user.

[1330] Output: Character data entered into the terminal.

[1331] Specific operation: The user inputs text data (e.g., "Thank you") using a device such as a smartphone or tablet.

[1332] Step 7:

[1333] The terminal transmits the input character data to the server.

[1334] Input: Character data entered by the user into a terminal.

[1335] Output: Character data sent to the server.

[1336] Specific operation: The terminal transmits the input character data to the server in real time via the network.

[1337] Step 8:

[1338] The server converts the character data into sign language motion data.

[1339] Input: Character data received by the server.

[1340] Output: The corresponding sign language motion data.

[1341] Specific operation: The server uses TextToSignModel to generate sign language motion data from character data.

[1342] Step 9:

[1343] The server renders the sign language motion data into a 3D animation format.

[1344] Input: Generated sign language motion data.

[1345] Output: Sign language motion in 3D animation format.

[1346] Specific operation: The server uses Render3DModel to convert sign language motion data into 3D animation.

[1347] Step 10:

[1348] The server sends the rendered 3D animation to a device capable of displaying it.

[1349] Input: Sign language motion in 3D animation format.

[1350] Output: Sign language animation displayed on smart glasses.

[1351] Specific operation: The server sends the generated 3D animation to the smart glasses via the network and displays it in real time.

[1352] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1353] This invention adds an emotion engine that recognizes the user's emotions to systems that translate sign language into text and systems that translate text into sign language, thereby improving the accuracy of sign language translation and realizing natural communication that takes emotions into account.

[1354] Translating sign language into text

[1355] 1. Acquiring video data

[1356] When the user signs, the device uses its camera to capture video of the sign.

[1357] The video data captured by the device is sent to the server in real time.

[1358] 2. Analysis of video data

[1359] The video data received by the server is divided into frames, and the position and shape of the hand in each frame is analyzed using a computer vision algorithm.

[1360] The server extracts the user's facial expressions from the video frames and recognizes the user's emotions using an emotion engine that uses facial recognition algorithms and natural language processing algorithms.

[1361] 3. Sign Language Identification

[1362] The server inputs the analysis results into a sign language recognition model (e.g., a deep learning model), which analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[1363] The system adjusts responses by combining recognized sign language with emotions inferred from the user's facial expressions.

[1364] 4. Conversion to Character Data

[1365] The server converts sign language into corresponding text, selecting an appropriate expression that reflects the user's emotions.

[1366] The server sends the converted string to the terminal and displays it to the user.

[1367] Example (translating sign language into text)

[1368] 1. The user smiles and signs "hello."

[1369] 2. The device uses its camera to capture sign language and facial expressions and sends the video to the server.

[1370] 3. The server divides the video into frames and analyzes the shape and position of the hands and facial expressions.

[1371] 4. The server uses the sign language recognition model to identify "hello" and the emotion engine recognizes a smile.

[1372] 5. The server generates the string "hello" and displays it in a friendly way.

[1373] 6. The terminal displays this string to the user.

[1374] Translating text into sign language

[1375] 1. Entering text data

[1376] The user inputs text data (e.g., "Thank you") into the terminal.

[1377] The terminal transmits the input character data to the server.

[1378] 2. Sign Language Generation

[1379] The server inputs the character data into a sign language generation algorithm to generate corresponding sign language motion data.

[1380] The server uses an emotion engine to predict facial expressions from input text and adjusts motion data based on that information.

[1381] 3. Sending sign language animations

[1382] The server renders the generated sign language motion data into a 3D animation format.

[1383] The server sends the rendered sign language animation data to the terminal.

[1384] 4. Displaying Sign Language Animation

[1385] The terminal plays back the received animation to the user and displays it as sign language motion that includes emotion.

[1386] Example (translating text into sign language)

[1387] 1. The user types the word "thank you" into the terminal.

[1388] 2. The terminal sends the character data to the server.

[1389] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[1390] 4. The server uses an emotion engine to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[1391] 5. The server renders this motion into a 3D animation and sends it back to the device.

[1392] 6. The device plays the animation and displays the sign language animation that reflects the emotion to the user.

[1393] This allows users to communicate naturally in real time, and conversion between sign language and text is performed in a way that includes emotional information.

[1394] The processing flow will be explained below.

[1395] Translating sign language into text

[1396] Step 1:

[1397] The user signs, using natural facial expressions and eye contact along with hand movements.

[1398] Step 2:

[1399] The device uses a camera to capture video of the user's sign language and facial expressions, and sends the captured video data to a server in real time.

[1400] Step 3:

[1401] The video data received by the server is divided into frames, which is a process of converting the video file into still image frames.

[1402] Step 4:

[1403] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts features of the hand shape, position, and movement.

[1404] Step 5:

[1405] The server analyzes the user's face in the video frames and uses facial recognition algorithms to recognize facial expressions, as well as natural language processing algorithms to infer emotions from facial expressions.

[1406] Step 6:

[1407] The server feeds the analyzed frames into a sign language recognition model (e.g., a deep learning model), which identifies the sign language words or phrases that correspond to the movements in each frame.

[1408] Step 7:

[1409] The server converts the sign language into the corresponding text based on the output of the sign language recognition model. At this time, the server adjusts the expression of the text taking into account the user's emotion estimated by the emotion engine.

[1410] Step 8:

[1411] The server sends the converted string data to the terminal.

[1412] Step 9:

[1413] The terminal displays the character string data received from the server to the user in an appropriate format that reflects the user's emotions.

[1414] Translating text into sign language

[1415] Step 1:

[1416] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[1417] Step 2:

[1418] The terminal sends the input or converted character data to the server.

[1419] Step 3:

[1420] The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data based on the text data.

[1421] Step 4:

[1422] The server uses an emotion engine to recognize the emotional tone from the input text data and voice data, and adjusts the sign language motion data based on the emotion estimated from the text data and voice data.

[1423] Step 5:

[1424] The server renders 3D animation based on the generated sign language motion data, which visualizes natural sign language motion that reflects emotional information.

[1425] Step 6:

[1426] The server sends the rendered sign language animation data to the terminal.

[1427] Step 7:

[1428] The terminal then plays back the received sign language animation and displays it to the user. This display is a natural sign language animation that reflects the user's emotions.

[1429] This allows users to easily translate between sign language and text in real time, incorporating emotions.

[1430] Example 2

[1431] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1432] When communicating between sign language users and other people, there is a need for a system that can accurately understand sign language and express emotions in real time. However, existing sign language translation systems have low sign language recognition accuracy and do not take into account the user's emotional expression, making natural communication difficult. To solve this problem, a system is needed that improves the accuracy of sign language recognition and translates the user's emotions.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1434] In this invention, the server includes means for capturing a video of a user's sign language movements, means for dividing the captured video into frames and analyzing the hand shape and position of each frame, means for extracting the user's facial expressions from the captured video frames and recognizing their emotions, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, and means for displaying the generated character string to the user. This enables highly accurate recognition of the user's sign language and natural translation that reflects the user's emotions.

[1435] "Means for capturing video of the actions of a sign language user" refers to devices such as cameras and sensors for recording the sign language actions of a sign language user, and includes technology for collecting those actions as digital video data.

[1436] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a software algorithm that breaks down the recorded video data into individual frames and identifies the position and shape of the user's hand in each frame. Specifically, it utilizes computer vision technology.

[1437] "Means for extracting a user's facial expressions from captured video frames and recognizing emotions" refers to a facial recognition algorithm and emotion analysis engine that analyzes facial features contained in video frames and grasps the user's emotional state.

[1438] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to the process of inputting the analyzed hand movement and shape data into a sign language recognition model (e.g., a model using a deep learning algorithm) and outputting the corresponding string of characters as a result.

[1439] "Means for displaying the generated character string to the user" refers to a device or software component for visually displaying the character string sent from the server to the user. Specifically, this includes the display of a smartphone or a dedicated display device.

[1440] "Means for converting character data into sign language motion data" refers to a software algorithm for converting input character string data into corresponding sign language motion data. Specifically, it includes a sign language dictionary database and a motion generation model.

[1441] "Means for rendering sign language motion data into a 3D animation format" refers to technology for rendering sign language motion data as 3D computer graphics to generate visually realistic sign language animation. Specifically, this refers to the use of 3D modeling software and rendering engines.

[1442] "Means for displaying the rendered animation to the user" refers to a video player component or display device that plays the generated 3D animation on the user's device, including a computer or mobile device display.

[1443] The present invention adds an emotion engine that recognizes the user's emotions to a system that translates sign language into text and a system that translates text into sign language. This improves the accuracy of sign language translation and realizes natural communication that reflects emotions. Specific embodiments of the present invention are described below.

[1444] Translating sign language into text

[1445] 1. Acquiring video data

[1446] The user signs.

[1447] The device (smartphone or camera-equipped device) records the sign language as a video, capturing the video in real time using the device's built-in camera.

[1448] The video data captured by the device is encoded in real time and sent to a server via the Internet using the RTMP protocol or WebRTC.

[1449] 2. Analysis of video data

[1450] The video data received by the server is divided into frames. This process uses a library such as FFmpeg.

[1451] The server analyzes each frame and uses OpenCV to identify the position and shape of the hand, using algorithms such as Edge Detection and Contour Detection.

[1452] The server uses a facial recognition algorithm (dlib, FaceNet, etc.) to extract the user's facial expressions from video frames, and then uses an emotion engine that utilizes natural language processing algorithms such as BERT to recognize the user's emotions.

[1453] 3. Sign Language Identification

[1454] The server inputs the analysis results into a sign language recognition model (a deep learning model built with TensorFlow or PyTorch), which analyzes the hand movements in each frame and identifies the corresponding sign language words and phrases.

[1455] The server combines the sign language recognition results with the emotions inferred by the emotion engine and adjusts the response.

[1456] 4. Conversion to Character Data

[1457] The server converts the sign language into a corresponding string of characters, choosing an appropriate expression that reflects the user's emotions. For example, for the sign "hello," the server converts it into "hello!", taking into account the smile.

[1458] The server encodes the generated character data and sends it to the terminal via the Internet.

[1459] The terminal decodes the received character data and displays it on the user's screen. Specifically, this is the operation of setting the character string in a GUI component.

[1460] Example (translating sign language into text)

[1461] The user smiles and signs "hello."

[1462] The device uses a camera to capture the sign language and facial expressions, and then sends the video to a server via the RTMP protocol.

[1463] The server splits the video into frames, analyzes hand shape and position using OpenCV, and analyzes facial expressions using dlib.

[1464] The server identifies "hello" using a TensorFlow model, and the emotion engine recognizes a smile using BERT.

[1465] The server generates the string "Hello!" and sends it to the terminal.

[1466] The terminal displays this string to the user.

[1467] Translating text into sign language

[1468] 1. Entering text data

[1469] The user enters text data (e.g., "Thank you") into the terminal, using a touchscreen or keyboard.

[1470] The device sends the input character data to the server via HTTP or WebSocket.

[1471] 2. Sign Language Generation

[1472] The server receives the character data and inputs it into a sign language generation algorithm (e.g., a deep learning model), which searches for corresponding sign language motion data from a character-to-sign language dictionary database.

[1473] The server uses an emotion engine to predict facial expressions from the input text and adjusts the sign language motion data based on that information. Specifically, it adjusts the movement and speed of the motion data to correspond to the emotion.

[1474] 3. Sending sign language animations

[1475] The server renders the generated sign language motion data using Blender or other 3D rendering software, and animates the 3D model of the sign language character.

[1476] The server encodes the 3D animation data and sends it to the device via the Internet.

[1477] 4. Displaying Sign Language Animation

[1478] The device decodes the received animation and plays it on the user's screen. Specifically, this is done by playing it using the video player component in the app.

[1479] The device displays emotional sign language motions to the user in 3D animation format.

[1480] Example (translating text into sign language)

[1481] The user types the word "thank you" into the device, for example, using the keyboard or touch screen of a smartphone.

[1482] The terminal sends the character data to the server via HTTP.

[1483] The server uses a sign language generation algorithm (TensorFlow model) to generate sign language motion data corresponding to "thank you."

[1484] The server uses an emotion engine (using the BERT model) to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[1485] The server uses Blender to render this motion data into a 3D animation, encodes it, and then sends it to the device.

[1486] The device decodes the animation and uses a video player to play the emotion-reflecting sign language animation on the user's screen.

[1487] Example prompts to input to the generative AI model

[1488] 1. Translating sign language to text: "When a user says 'hello' in sign language, analyze their hand movements and facial expressions and generate an appropriate string of text based on their emotions."

[1489] 2. Example of translating text into sign language: "Generate motion data to express the word 'thank you' in sign language, and create a 3D animation that reflects the emotion of gratitude."

[1490] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1491] Translating sign language into text

[1492] Step 1:

[1493] The user signs

[1494] Input: User's sign language actions

[1495] Specific action: The user signs to the camera, for example, signing "hello."

[1496] Output: Video data including the user's sign language

[1497] Step 2:

[1498] The device captures video and sends it to the server in real time.

[1499] Input: Video data containing the user's sign language

[1500] Specific operation: The system uses the device's camera to capture sign language video in real time, encodes the video data using the RTMP protocol, and then transmits the encoded video data to a server via the Internet.

[1501] Output: Video data sent to the server

[1502] Step 3:

[1503] The server divides the video data into frames and analyzes the shape and position of the hand.

[1504] Input: Video data

[1505] How it works: The server splits the video data into frames using the FFmpeg library. For each frame, it analyzes the hand shape and position using the Edge Detection and Contour Detection algorithms using OpenCV.

[1506] Output: Analysis results of hand shape and position

[1507] Step 4:

[1508] The server extracts facial expressions from video frames and recognizes emotions.

[1509] Input: Video data

[1510] Specific operation: The server uses dlib and FaceNet to detect facial features from frames, and uses BERT, a natural language processing algorithm, to recognize the user's emotions from the detected facial features.

[1511] Output: User emotion data

[1512] Step 5:

[1513] The server inputs the analysis results into a sign language recognition model and generates the corresponding string.

[1514] Input: Analysis results of hand shape and position, user emotion data

[1515] How it works: The server inputs the analysis results into a deep learning model built with TensorFlow or PyTorch, converts the sign language gestures into corresponding text, and uses the emotion data to select the appropriate expression method as needed.

[1516] Output: The corresponding string

[1517] Step 6:

[1518] The server sends the generated string to the terminal, which displays the string to the user.

[1519] Input: The corresponding string

[1520] Specific operation: The server encodes the generated string and sends it to the terminal via the Internet. The terminal decodes the received string and displays it in the GUI component.

[1521] Output: The string displayed on the user's screen

[1522] Translating text into sign language

[1523] Step 1:

[1524] The user enters character data

[1525] Input: Text data (e.g. "Thank you")

[1526] Specific Action: A user types text using the keyboard or touchscreen of a smartphone or tablet.

[1527] Output: Character data entered

[1528] Step 2:

[1529] The terminal sends the input character data to the server.

[1530] Input: Character data entered

[1531] Specific operation: The terminal sends character data to the server using HTTP or WebSocket.

[1532] Output: Character data sent to the server

[1533] Step 3:

[1534] The server converts the text data into sign language motion data.

[1535] Input: Character data

[1536] How it works: The server inputs character data into a sign language generation algorithm (e.g., a TensorFlow model) to generate sign language motion data. It then uses an emotion engine to infer emotions from the character data and adjusts the motion data based on the results.

[1537] Output: Sign language motion data

[1538] Step 4:

[1539] The server renders the sign language motion data into a 3D animation format.

[1540] Input: Sign language motion data

[1541] Specific operation: The server uses 3D rendering software such as Blender to render the sign language motion data into 3D animation.

[1542] Output: 3D animation data

[1543] Step 5:

[1544] The server sends 3D animation data to the terminal, and the terminal displays the animation to the user.

[1545] Input: 3D animation data

[1546] Specific operation: The server encodes 3D animation data and sends it to the device via the Internet. The device decodes the received animation data and plays it using the video player component.

[1547] Output: The animation played on the user's screen

[1548] (Application example 2)

[1549] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1550] Conventional sign language translation systems simply convert sign language into text, making it difficult to achieve natural communication that reflects the user's emotions. Furthermore, they lacked the ability to output text generated from sign language as speech, or the ability to convert text into sign language animation. This made it difficult for hearing-impaired people and people who use sign language to communicate smoothly on public transportation, such as self-driving vehicles.

[1551] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a video of a user's actions using sign language, means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, means for extracting the user's facial expression from the video data and recognizing the emotion using an emotion engine, means for adjusting the character string to reflect the user's emotion, means for outputting the generated character string as audio, means for converting character data into sign language motion data, means for rendering the sign language motion data in a 3D animation format, and means for reflecting emotion in the rendered animation using the emotion engine. This enables users to communicate naturally and emotionally even inside an autonomous vehicle.

[1552] 1. "Means for capturing the actions of a user using sign language as video" refers to a device or system that records the actions of a user using sign language in video format using an image capture device such as a camera.

[1553] 2. "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to an algorithm or software that breaks down video data into individual frames and analyzes the shape and position of the user's hand within each frame.

[1554] 3. "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to a process or system that inputs the analyzed hand movement data into a sign language recognition model and generates a corresponding string of characters as a result.

[1555] 4. "Means for displaying the generated character string to the user" refers to a device or system that outputs the generated character data to a display or other display device and presents it in a form that the user can visually confirm.

[1556] 5. "Means for extracting a user's facial expressions from video data and recognizing emotions using an emotion engine" means a process or system that extracts a user's facial expressions from video data and identifies the user's emotional state using machine learning or artificial intelligence algorithms.

[1557] 6. "Means for adjusting text to reflect user emotions" refers to an algorithm or process that appropriately changes the content and expression of generated text based on recognized user emotional information.

[1558] 7. "Means for audibly outputting the generated character string" refers to a device or system that uses speech synthesis technology to play back the generated character data as speech.

[1559] 8. "Means for converting character data into sign language motion data" means an algorithm or software that converts input character information into corresponding sign language motion data.

[1560] 9. "Means for rendering sign language motion data into a 3D animation format" means a process or system that renders the generated sign language motion data as a three-dimensional animation.

[1561] 10. "Means for reflecting emotions in rendered animation using an emotion engine" means a system or algorithm that applies an emotion engine to generated 3D animation and reflects emotions in the character's movements and expressions.

[1562] This invention relates to a system that enables users who use sign language to communicate naturally and emotionally within an autonomous vehicle. This system captures sign language movements as video and generates text, audio, and sign language animations based on the analysis results, thereby supporting comfortable communication for users.

[1563] System Program

[1564] The system uses the following hardware and software:

[1565] Hardware: In-vehicle camera, high-resolution display, on-board computer, audio output device

[1566] Software: Computer vision libraries (e.g., OpenCV), deep learning frameworks (e.g., TensorFlow, PyTorch), natural language processing libraries (e.g., NLTK, Transformers), emotion recognition engines

[1567] System action

[1568] 1. Acquiring video data:

[1569] The in-vehicle camera captures the user's sign language in real time as video, which is then broken down into frames and the shape and position of the hand in each frame are analyzed.

[1570] 2. Sign Language Analysis and Recognition:

[1571] The analyzed data is input into a sign language recognition model (deep learning algorithm) to generate a corresponding string of characters. At the same time, the user's facial expressions are extracted from the video data, and an emotion engine recognizes the user's emotions.

[1572] 3. Creating and displaying strings:

[1573] The system adjusts the text based on the recognized sign language and emotional information, and the resulting text is displayed on the vehicle's display. The text is also output as speech using speech synthesis technology.

[1574] 4. Generating Sign Language Animation:

[1575] Conversely, text data entered by the user is converted into sign language motion data, which is then rendered into a 3D animation format and emotion-based using an emotion recognition engine. The final rendered animation is then displayed on the screen.

[1576] Specific examples

[1577] Translating sign language into text and speech

[1578] 1. A user signs "thank you" inside a self-driving vehicle.

[1579] 2. Cameras inside the vehicle capture sign language in real time and transmit the data to an onboard computer.

[1580] 3. The on-board computer divides the video into frames and analyzes hand movements and the user's facial expressions.

[1581] 4. Based on the analysis results, the sign language recognition model and emotion engine generate the string "Thank you" and emotional information.

[1582] 5. "Thank you" will appear on the display and be output as audio at the same time.

[1583] Translating text into sign language animation

[1584] 1. The user types "Help me" into the display.

[1585] 2. The on-board computer converts the text data into sign language motion data, and the emotion engine recognizes and reflects the emotion of "urgency."

[1586] 3. The generated sign language motion data is rendered as a 3D animation and displayed on the screen.

[1587] Prompt Sentence Examples

[1588] Translate sign language to text: A user signs "Where is my house?" You can see confusion in the user's face.

[1589] Information from sign language:

[1590] Transcription: Where is my house?

[1591] Emotion: Confused

[1592] Translate text into sign language: The user has typed "Thank you." Reflect the sentiment of gratitude with the emotion engine.

[1593] Generated sign language motion:

[1594] Sign Language: Thank you

[1595] Emotion: Gratitude

[1596] This will enable users to have natural and emotionally rich communication even inside an autonomous vehicle.

[1597] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1598] Step 1:

[1599] Acquiring video data using a camera: When a user signs, the device's in-car camera captures the action in real time. This captures video data including the user's hand movements and facial expressions. The input is "video data including the user's sign language and facial expressions," and the output is "video data acquired in real time."

[1600] Step 2:

[1601] Video data frame division and analysis: The server receives the acquired video data and divides the video into frames. The shape and position of the hand in each frame are analyzed using a computer vision library (e.g., OpenCV). The input is "video data acquired in real time," and the output is "hand shape and position data for each frame."

[1602] Step 3:

[1603] Sign language recognition: The server inputs the analyzed frame data into a sign language recognition model (e.g., a deep learning algorithm using TensorFlow or PyTorch) to identify the corresponding sign language words or phrases. The input is the hand shape and position data for each frame, and the output is the identified sign language string.

[1604] Step 4:

[1605] Facial expression analysis and emotion recognition: The server extracts the user's facial expressions from the video data and recognizes the user's emotions using a natural language processing algorithm and an emotion engine. The input is "video data acquired in real time" and the output is "user's emotional information."

[1606] Step 5:

[1607] Generating and displaying text and audio: The server adjusts the text based on the recognized sign language text and emotional information, and displays it on the vehicle's display. The text is also output as audio using a speech synthesis engine. The input is the "identified sign language text and the user's emotional information," and the output is "text and audio data displayed to the user."

[1608] Step 6:

[1609] Conversion of character data into sign language motion data: The server receives character data entered by the user and converts it into corresponding sign language motion data. The input is "character data entered by the user" and the output is "sign language motion data."

[1610] Step 7:

[1611] Rendering sign language motion data and reflecting emotions: The server renders the sign language motion data into 3D animation format and reflects emotional information using an emotion engine. The input is "sign language motion data" and "emotional information," and the output is "emotionally reflected sign language 3D animation."

[1612] Step 8:

[1613] Displaying sign language animation: The device displays the rendered sign language 3D animation on the display, providing a visual experience to the user. The input is "emotion-reflecting sign language 3D animation," and the output is "sign language 3D animation displayed to the user."

[1614] Through these steps, users can communicate naturally and emotively using sign language inside a self-driving vehicle.

[1615] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1616] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1617] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1618] [Fourth embodiment]

[1619] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1620] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1621] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1622] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1623] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1624] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1625] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1626] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1627] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1628] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1629] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1630] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1631] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1632] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[1633] Translating sign language into text

[1634] 1. Acquiring video data

[1635] When the user signs, the device uses its camera to capture video of the sign.

[1636] The captured video data is sent to a server in real time.

[1637] 2. Analysis of video data

[1638] The video received by the server is divided into frames.

[1639] The hand position and shape in each frame are analyzed using a computer vision algorithm (e.g., OpenCV).

[1640] 3. Sign Language Identification

[1641] The server inputs the analyzed data into a sign language recognition model (e.g., a deep learning model).

[1642] The sign language recognition model analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[1643] 4. Conversion to Character Data

[1644] The server converts the identified sign language into a corresponding string.

[1645] The converted string is sent to the terminal, which displays the characters to the user.

[1646] Example (translating sign language into text)

[1647] 1. The user signs "hello."

[1648] 2. The device captures the sign language with its camera and sends the video to the server.

[1649] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[1650] 4. The server uses the recognition model to determine that this motion is a "hello."

[1651] 5. The server generates the string "Hello" and sends it to the terminal.

[1652] 6. The terminal displays this string to the user.

[1653] Translating text into sign language

[1654] 1. Entering text data

[1655] The user inputs text data (for example, "Thank you") into the terminal.

[1656] In the case of a live voice, the audio is recorded using the device's microphone and the audio data is converted into text.

[1657] The text data is sent to the server.

[1658] 2. Sign Language Generation

[1659] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data.

[1660] Once the motion data is generated, the server renders it into a 3D animation format.

[1661] 3. Sending sign language animations

[1662] The server transmits the generated sign language animation data to the terminal.

[1663] Users can view sign language animations on their devices.

[1664] Example (translating text into sign language)

[1665] 1. The user types the word "thank you" into the terminal.

[1666] 2. The terminal sends the character data to the server.

[1667] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[1668] 4. The server renders this motion into a 3D animation and sends it back to the device.

[1669] 5. The device plays the animation and displays it to the user.

[1670] This allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motion, enabling smooth communication with the hearing impaired.

[1671] The processing flow will be explained below.

[1672] Translating sign language into text

[1673] Step 1:

[1674] The user signs a particular phrase or word.

[1675] Step 2:

[1676] The device uses a camera to record the user's sign language as a video, and the video data is sent to a server in real time.

[1677] Step 3:

[1678] The video data received by the server is divided into frames. This division is the process of converting the video into individual still images (frames).

[1679] Step 4:

[1680] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts data on the hand's shape, position, and movement characteristics.

[1681] Step 5:

[1682] The server feeds the parsed frames into a sign language recognition model (e.g., a deep learning model), which identifies sign language words and phrases based on hand movements and position information.

[1683] Step 6:

[1684] The server converts the signs that are identified with high confidence into corresponding strings of characters based on the output from the sign language recognition model.

[1685] Step 7:

[1686] The server sends the converted string data to the terminal.

[1687] Step 8:

[1688] The terminal displays the character string data received from the server to the user.

[1689] Translating text into sign language

[1690] Step 1:

[1691] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[1692] Step 2:

[1693] The terminal transmits the input character data to the server.

[1694] Step 3:

[1695] The server inputs the text data into a sign language generation algorithm, which converts the text data into sign language motion data.

[1696] Step 4:

[1697] The server renders 3D animation based on the generated sign language motion data. During this rendering process, the sign language motion is visually represented.

[1698] Step 5:

[1699] The server sends the rendered sign language animation data to the terminal.

[1700] Step 6:

[1701] The terminal plays the received sign language animation and displays it to the user.

[1702] This allows users to input sign language and convert it into text, and input text or voice and convert it into sign language in real time.

[1703] Example 1

[1704] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1705] Conventional sign language translation systems have had difficulty accurately recognizing sign language movements and converting them into text in real time. Furthermore, when converting text data into sign language motion, there was a lack of technology to generate natural-looking sign language animations. As a result, smooth communication with the hearing impaired was difficult.

[1706] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1707] In this invention, the server includes means for capturing user actions as video, means for dividing the captured video into frames and analyzing the hand shape and position in each frame, means for inputting the analysis results into a sign language recognition model and converting the sign language actions into corresponding character strings, and means for displaying the generated character strings to the user in real time, thereby enabling the sign language actions to be accurately and quickly converted into character strings and displayed to the user.

[1708] In this invention, the server further includes means for converting character data into sign language motion data, means for rendering the sign language motion data into a 3D animation format, and means for displaying the rendered animation to the user, thereby enabling input character data to be displayed as natural sign language animation.

[1709] "User" refers to a person who uses sign language or inputs character data.

[1710] "Terminal" refers to a device that includes a camera that captures video of the user signing and a device that inputs text data.

[1711] "Server" refers to a computer system that receives and analyzes video and text data sent from a terminal, and performs sign language recognition and generates 3D animation.

[1712] "Video data" refers to video information captured by a camera of a user's sign language actions.

[1713] A "frame" refers to each still image that makes up video data.

[1714] "Computer vision algorithm" refers to image processing technology for analyzing the position and shape of hands from video data.

[1715] "Sign language recognition model" refers to a deep learning model for identifying sign language words and phrases based on analyzed hand movements.

[1716] "Character string" refers to text information in which sign language actions are converted into corresponding characters.

[1717] "Sign language motion data" refers to data that expresses sign language movements and is generated based on character data.

[1718] "3D animation format" refers to a format that expresses sign language motion data as three-dimensional movement.

[1719] "Real-time" means that the system processes and converts data instantly and displays the results without delay.

[1720] The present invention provides a system for converting sign language into text in real time, and a system for converting text into sign language. The system includes a process for acquiring and analyzing the movements of a sign language user, converting identified signs into corresponding text, and converting text and voice data into sign language motion data, which is then displayed in a 3D animation format.

[1721] Translating sign language into text

[1722] When a user signs, the device uses a camera to capture video of the sign language. Specifically, a camera installed on a smartphone or PC is used. The captured video data is sent to a server in real time. The server divides the received video into frames and analyzes the position and shape of the hands in each frame. This analysis is performed using computer vision algorithms such as OpenCV. The analyzed data is input into a sign language recognition model. A deep learning model built using TensorFlow or PyTorch is used as the sign language recognition model. The recognition model analyzes the hand movements in each frame and identifies the corresponding sign language word or phrase. The identified sign language is converted into the corresponding string of characters, and the converted string is sent to the device. The device then displays the characters to the user.

[1723] For example, if a user signs "hello," the device captures the sign with a camera and sends the video to the server. The server divides the video into frames and recognizes the shape and position of the hand. The server then uses a recognition model to determine that this movement is "hello." The server generates the string "hello" and sends it to the device. The device then displays this string to the user.

[1724] Translating text into sign language

[1725] The user inputs text data (for example, "Thank you") into the device. If the input is human voice, the audio is recorded using the device's microphone, and the audio data is converted into text using a method such as the Google Cloud Speech-to-Text API. This text data is sent to the server. The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data. The generated motion data is rendered in a 3D animation format. Rendering is done using Unity or Blender. The generated sign language animation data is sent to the device, and the user can view the sign language animation on the device.

[1726] For example, if a user types the character "Thank you" into a device, the device sends the character data to the server. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "Thank you." The server renders this motion into a 3D animation and sends it back to the device. The device plays the animation and displays it to the user.

[1727] This system allows users to easily obtain information in text form simply by using sign language, and the input text and voice data is displayed as natural sign language motions, enabling smooth communication with the hearing impaired.

[1728] Prompt Sentence Examples

[1729] "Please explain how this system translates sign language into text in real time and simultaneously renders sign language animations in multiple dimensions."

[1730] Hardware and software used

[1731] Hardware: smartphone, computer, camera, microphone

[1732] Software: OpenCV, TensorFlow, PyTorch, Google Cloud Speech-to-Text API, Unity, Blender

[1733] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1734] Step 1:

[1735] Acquiring video data

[1736] Input: User's sign language actions

[1737] The device captures the user's sign language movements with a camera, which can be a built-in camera on a smartphone or PC, and transmits the captured video data to a server in real time.

[1738] Step 2:

[1739] Video data analysis

[1740] Input: Video data from the device

[1741] Output: Hand position and shape data per frame

[1742] The server splits the received video into frames using a library such as FFmpeg. The position and shape of the hand in each frame is analyzed using a computer vision algorithm (e.g., OpenCV). For example, the position of the hand can be identified by detecting a specific color area within the frame.

[1743] Step 3:

[1744] Sign Language Identification

[1745] Input: Parsed frame data

[1746] Output: A string corresponding to the sign language action

[1747] The server inputs the parsed data into a sign language recognition model, which uses deep learning models built with TensorFlow and PyTorch. The model analyzes the hand movements in each frame and identifies which sign language words or phrases they correspond to. For example, a series of hand movements may have a pattern that corresponds to a specific sign language word.

[1748] Step 4:

[1749] Conversion to character data

[1750] Input: Recognition results of sign language recognition model

[1751] Output: The corresponding string

[1752] The server converts the sign language actions identified by the sign language recognition model into a corresponding string. For example, if the hand movement is identified as corresponding to "hello," the server generates the string "hello." The server then transmits the generated string to the terminal.

[1753] Step 5:

[1754] Displaying a string

[1755] Input: The string sent from the server

[1756] Output: The string the user sees on the terminal

[1757] The terminal displays the text sent from the server on the screen in real time using a standard text drawing library.

[1758] Step 6:

[1759] Entering character data

[1760] Input: Character data entered by the user.

[1761] The user inputs text data (for example, "Thank you") into the device. In the case of voice input, the voice is recorded with the device's microphone and converted into text using the Google Cloud Speech-to-Text API or similar. The text data is then sent to the server.

[1762] Step 7:

[1763] Generating sign language motion data

[1764] Input: Character data

[1765] Output: Sign language motion data

[1766] The server inputs the text data into a sign language generation algorithm to generate corresponding sign language motion data, which expresses the sign language movements in detail.

[1767] Step 8:

[1768] 3D animation rendering

[1769] Input: Sign language motion data

[1770] Output: 3D animation data

[1771] The server renders the sign language motion data into a 3D animation format using Unity or Blender to visually represent the motion data.

[1772] Step 9:

[1773] Playing and viewing animations

[1774] Input: 3D animation data

[1775] Output: Sign language animation that the user sees on their device

[1776] The server sends the generated 3D animation data to the device, which then plays the animation and displays it to the user, allowing the user to see the sign language through the animation.

[1777] (Application example 1)

[1778] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1779] Conventional sign language recognition systems can convert sign language into text and vice versa, but they lack the support functions to facilitate smooth face-to-face communication in brick-and-mortar stores, etc. In particular, there is a demand for a system that uses smart glasses to convert between sign language and text in real time and display sign language motion as 3D animation, enabling smooth communication with the hearing impaired.

[1780] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1781] In this invention, the server includes: means for capturing a video of a user's sign language movements; means for dividing the captured video into frames and analyzing the hand shape and position of each frame; means for inputting the analysis results into a sign language recognition model and generating a corresponding character string; means for displaying the generated character string on a displayable device; means for displaying the generated character string on smart glasses; means for converting character data into sign language motion data; means for rendering the sign language motion data into a 3D animation format; and means for displaying the rendered animation on a displayable device. This enables real-time conversion between sign language and characters using smart glasses in face-to-face communication such as in brick-and-mortar stores, enabling smooth communication with hearing-impaired people.

[1782] "Means for capturing the actions of a user performing sign language as video" refers to a device that uses a camera or other sensor to record the sign language movements of a user as video data.

[1783] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a system that divides the recorded video into frames and uses algorithms and computer vision technology to analyze the position and shape of the hand in each frame.

[1784] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding character string" refers to the process of inputting the analysis results of hand position and shape into a sign language recognition model and generating a corresponding character string as a result. This sign language recognition model utilizes technologies such as machine learning and deep learning.

[1785] The "means for presenting the generated character string on a displayable device" refers to a method for outputting the generated character string to a display device such as a smartphone, tablet, or computer display.

[1786] The "means for displaying the generated character string on the smart glasses" refers to a device that displays the generated character string on the display of the smart glasses in real time. When a user wears the smart glasses, they can simultaneously view the real world and the display.

[1787] "Means for converting character data into sign language motion data" refers to an algorithm or computer program for converting input character data into sign language motion, thereby converting the character information into dynamic sign language expressions.

[1788] "Means for rendering sign language motion data into a 3D animation format" refers to a process or system that generates 3D animation from sign language motion data. This rendering creates a visually realistic 3D animation of the sign language motion.

[1789] A "means for presenting rendered animation on a display-capable device" refers to a method or apparatus for showing the generated 3D animation to a user on a smartphone, tablet, computer display, etc.

[1790] The present invention relates to a system for converting sign language into text in real time, and a system for converting text into sign language. This system has the function of acquiring and analyzing a user's sign language movements and converting them into text, and conversely, converting the text into sign language motion data and displaying it. Specific embodiments of the present invention will be described below.

[1791] Hardware and Software Configuration

[1792] This system is implemented mainly using the following hardware and software.

[1793] Hardware: Smart glasses (with camera and display capabilities), server, displayable devices (smartphones, tablets, computers, etc.)

[1794] software:

[1795] Computer vision algorithms: Image processing libraries such as OpenCV

[1796] Sign language recognition model library: HandSignRecognitionModel

[1797] A library for converting text to sign language motion: TextToSignModel

[1798] 3D animation rendering library: Render3DModel

[1799] Overview of data processing and calculation

[1800] 1. Acquiring video data

[1801] The server uses a camera to record the sign language actions performed by the user. The camera in the smart glasses captures the sign language in real time and sends the video data to the server.

[1802] 2. Analysis of video data

[1803] The acquired video data is split into frames on the server, and computer vision algorithms such as OpenCV are used to analyze the shape and position of the hand in each frame.

[1804] 3. Sign Language Identification

[1805] The analysis results of hand shape and position are input into a sign language recognition model (HandSignRecognitionModel), which uses a deep learning algorithm to analyze the hand movements in each frame and identify the corresponding sign language word or phrase.

[1806] 4. Conversion to Character Data

[1807] The sign language identified by the sign language recognition model is converted into the corresponding text on the server and sent to a display-capable device or smart glasses, which then display the text information on the display in real time.

[1808] 5. Converting character data into sign language motion data

[1809] The server inputs character data from the user into a TextToSignModel and generates corresponding sign language motion data, which is then rendered into a 3D animation format using a Render3DModel.

[1810] 6. Displaying Sign Language Animation

[1811] The generated 3D animation is sent from the server to smart glasses or other display-capable devices and displayed in real time.

[1812] Specific operation example

[1813] Example 1: Sign language to text conversion

[1814] 1. The user signs "hello."

[1815] 2. The camera in the smart glasses captures the sign language and sends the video to a server.

[1816] 3. The server divides the video into frames and recognizes the shape and position of the hand.

[1817] 4. The server uses a sign language recognition model to identify this movement as "hello."

[1818] 5. The server generates the string "hello" and sends it to the smart glasses.

[1819] 6. The smart glasses will display this string in real time.

[1820] Example 2: Converting text to sign language

[1821] 1. The user types "thank you" into their smartphone.

[1822] 2. The entered character data is sent to the server.

[1823] 3. The server uses TextToSignModel to generate sign language motion data corresponding to "thank you."

[1824] 4. The server renders this motion into a 3D animation and sends it to the smart glasses.

[1825] 5. The smart glasses play the animation and display it to the user.

[1826] Prompt Sentence Examples

[1827] Examples of prompts for generative AI models include:

[1828] prompt:

[1829] A user signs "Hello." This video is captured by a camera and sent to a sign language recognition system. The system uses a sign language recognition model to analyze the movement and generate the corresponding text "Hello." This text should then be displayed on the smart glasses.

[1830] output:

[1831] The text "Hello" will appear on the smartglasses display.

[1832] This will enable real-time conversion between sign language and text using smart glasses during face-to-face communication in brick-and-mortar stores, enabling smooth communication with the hearing impaired.

[1833] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1834] Step 1:

[1835] The user signs.

[1836] Input: The user's sign language action.

[1837] Output: Video data captured by the camera.

[1838] Specific operation: The camera on the device (smart glasses) captures the user's sign language movements in real time and acquires them as video data.

[1839] Step 2:

[1840] The video data acquired by the terminal is sent to the server.

[1841] Input: Video data captured by the camera.

[1842] Output: Video data sent to the server.

[1843] Specific operation: The network connection is used to send the video data acquired by the device to the server in real time.

[1844] Step 3:

[1845] The server divides the video data into frames and analyzes the shape and position of the hand in each frame.

[1846] Input: Video data received by the server.

[1847] Output: Analysis results of hand shape and position for each frame.

[1848] How it works: The server divides the video data into frames and uses computer vision algorithms such as OpenCV to identify the shape and position of the hand for each frame.

[1849] Step 4:

[1850] The server inputs the analysis results into a sign language recognition model (HandSignRecognitionModel) and generates a corresponding string.

[1851] Input: Analysis results of hand shape and position.

[1852] Output: The corresponding string data.

[1853] How it works: The server inputs hand shape and position data into a sign language recognition model, then uses a deep learning algorithm to analyze the sign language and generate a corresponding string of characters.

[1854] Step 5:

[1855] The server sends the generated string to a display-capable device or smart glasses.

[1856] Input: String data generated by a sign language recognition model.

[1857] Output: The string to be displayed on the smart glasses.

[1858] Specific operation: The server sends the generated string data to the smart glasses via the network and displays it in real time.

[1859] Step 6:

[1860] The user inputs character data.

[1861] Input: Character data entered by the user.

[1862] Output: Character data entered into the terminal.

[1863] Specific operation: The user inputs text data (e.g., "Thank you") using a device such as a smartphone or tablet.

[1864] Step 7:

[1865] The terminal transmits the input character data to the server.

[1866] Input: Character data entered by the user into a terminal.

[1867] Output: Character data sent to the server.

[1868] Specific operation: The terminal transmits the input character data to the server in real time via the network.

[1869] Step 8:

[1870] The server converts the character data into sign language motion data.

[1871] Input: Character data received by the server.

[1872] Output: The corresponding sign language motion data.

[1873] Specific operation: The server uses TextToSignModel to generate sign language motion data from character data.

[1874] Step 9:

[1875] The server renders the sign language motion data into a 3D animation format.

[1876] Input: Generated sign language motion data.

[1877] Output: Sign language motion in 3D animation format.

[1878] Specific operation: The server uses Render3DModel to convert sign language motion data into 3D animation.

[1879] Step 10:

[1880] The server sends the rendered 3D animation to a device capable of displaying it.

[1881] Input: Sign language motion in 3D animation format.

[1882] Output: Sign language animation displayed on smart glasses.

[1883] Specific operation: The server sends the generated 3D animation to the smart glasses via the network and displays it in real time.

[1884] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1885] This invention adds an emotion engine that recognizes the user's emotions to systems that translate sign language into text and systems that translate text into sign language, thereby improving the accuracy of sign language translation and realizing natural communication that takes emotions into account.

[1886] Translating sign language into text

[1887] 1. Acquiring video data

[1888] When the user signs, the device uses its camera to capture video of the sign.

[1889] The video data captured by the device is sent to the server in real time.

[1890] 2. Analysis of video data

[1891] The video data received by the server is divided into frames, and the position and shape of the hand in each frame is analyzed using a computer vision algorithm.

[1892] The server extracts the user's facial expressions from the video frames and recognizes the user's emotions using an emotion engine that uses facial recognition algorithms and natural language processing algorithms.

[1893] 3. Sign Language Identification

[1894] The server inputs the analysis results into a sign language recognition model (e.g., a deep learning model), which analyzes the hand movements in each frame and identifies the corresponding sign language words or phrases.

[1895] The system adjusts responses by combining recognized sign language with emotions inferred from the user's facial expressions.

[1896] 4. Conversion to Character Data

[1897] The server converts sign language into corresponding text, selecting an appropriate expression that reflects the user's emotions.

[1898] The server sends the converted string to the terminal and displays it to the user.

[1899] Example (translating sign language into text)

[1900] 1. The user smiles and signs "hello."

[1901] 2. The device uses its camera to capture sign language and facial expressions and sends the video to the server.

[1902] 3. The server divides the video into frames and analyzes the shape and position of the hands and facial expressions.

[1903] 4. The server uses the sign language recognition model to identify "hello" and the emotion engine recognizes a smile.

[1904] 5. The server generates the string "hello" and displays it in a friendly way.

[1905] 6. The terminal displays this string to the user.

[1906] Translating text into sign language

[1907] 1. Entering text data

[1908] The user inputs text data (e.g., "Thank you") into the terminal.

[1909] The terminal transmits the input character data to the server.

[1910] 2. Sign Language Generation

[1911] The server inputs the character data into a sign language generation algorithm to generate corresponding sign language motion data.

[1912] The server uses an emotion engine to predict facial expressions from input text and adjusts motion data based on that information.

[1913] 3. Sending sign language animations

[1914] The server renders the generated sign language motion data into a 3D animation format.

[1915] The server sends the rendered sign language animation data to the terminal.

[1916] 4. Displaying Sign Language Animation

[1917] The terminal plays back the received animation to the user and displays it as sign language motion that includes emotion.

[1918] Example (translating text into sign language)

[1919] 1. The user types the word "thank you" into the terminal.

[1920] 2. The terminal sends the character data to the server.

[1921] 3. The server uses a sign language generation algorithm to generate sign language motion data corresponding to "thank you."

[1922] 4. The server uses an emotion engine to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[1923] 5. The server renders this motion into a 3D animation and sends it back to the device.

[1924] 6. The device plays the animation and displays the sign language animation that reflects the emotion to the user.

[1925] This allows users to communicate naturally in real time, and conversion between sign language and text is performed in a way that includes emotional information.

[1926] The processing flow will be explained below.

[1927] Translating sign language into text

[1928] Step 1:

[1929] The user signs, using natural facial expressions and eye contact along with hand movements.

[1930] Step 2:

[1931] The device uses a camera to capture video of the user's sign language and facial expressions, and sends the captured video data to a server in real time.

[1932] Step 3:

[1933] The video data received by the server is divided into frames, which is a process of converting the video file into still image frames.

[1934] Step 4:

[1935] The server analyzes the hand position and shape in each frame using a computer vision algorithm (e.g., OpenCV), which extracts features of the hand shape, position, and movement.

[1936] Step 5:

[1937] The server analyzes the user's face in the video frames and uses facial recognition algorithms to recognize facial expressions, as well as natural language processing algorithms to infer emotions from facial expressions.

[1938] Step 6:

[1939] The server feeds the analyzed frames into a sign language recognition model (e.g., a deep learning model), which identifies the sign language words or phrases that correspond to the movements in each frame.

[1940] Step 7:

[1941] The server converts the sign language into the corresponding text based on the output of the sign language recognition model. At this time, the server adjusts the expression of the text taking into account the user's emotion estimated by the emotion engine.

[1942] Step 8:

[1943] The server sends the converted string data to the terminal.

[1944] Step 9:

[1945] The terminal displays the character string data received from the server to the user in an appropriate format that reflects the user's emotions.

[1946] Translating text into sign language

[1947] Step 1:

[1948] The user inputs text data into the terminal, or records voice using the terminal's microphone and converts the voice data into text.

[1949] Step 2:

[1950] The terminal sends the input or converted character data to the server.

[1951] Step 3:

[1952] The server inputs the text data into a sign language generation algorithm, which generates corresponding sign language motion data based on the text data.

[1953] Step 4:

[1954] The server uses an emotion engine to recognize the emotional tone from the input text data and voice data, and adjusts the sign language motion data based on the emotion estimated from the text data and voice data.

[1955] Step 5:

[1956] The server renders 3D animation based on the generated sign language motion data, which visualizes natural sign language motion that reflects emotional information.

[1957] Step 6:

[1958] The server sends the rendered sign language animation data to the terminal.

[1959] Step 7:

[1960] The terminal then plays back the received sign language animation and displays it to the user. This display is a natural sign language animation that reflects the user's emotions.

[1961] This allows users to easily translate between sign language and text in real time, incorporating emotions.

[1962] Example 2

[1963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1964] When communicating between sign language users and other people, there is a need for a system that can accurately understand sign language and express emotions in real time. However, existing sign language translation systems have low sign language recognition accuracy and do not take into account the user's emotional expression, making natural communication difficult. To solve this problem, a system is needed that improves the accuracy of sign language recognition and translates the user's emotions.

[1965] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1966] In this invention, the server includes means for capturing a video of a user's sign language movements, means for dividing the captured video into frames and analyzing the hand shape and position of each frame, means for extracting the user's facial expressions from the captured video frames and recognizing their emotions, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, and means for displaying the generated character string to the user. This enables highly accurate recognition of the user's sign language and natural translation that reflects the user's emotions.

[1967] "Means for capturing video of the actions of a sign language user" refers to devices such as cameras and sensors for recording the sign language actions of a sign language user, and includes technology for collecting those actions as digital video data.

[1968] "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to a software algorithm that breaks down the recorded video data into individual frames and identifies the position and shape of the user's hand in each frame. Specifically, it utilizes computer vision technology.

[1969] "Means for extracting a user's facial expressions from captured video frames and recognizing emotions" refers to a facial recognition algorithm and emotion analysis engine that analyzes facial features contained in video frames and grasps the user's emotional state.

[1970] "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to the process of inputting the analyzed hand movement and shape data into a sign language recognition model (e.g., a model using a deep learning algorithm) and outputting the corresponding string of characters as a result.

[1971] "Means for displaying the generated character string to the user" refers to a device or software component for visually displaying the character string sent from the server to the user. Specifically, this includes the display of a smartphone or a dedicated display device.

[1972] "Means for converting character data into sign language motion data" refers to a software algorithm for converting input character string data into corresponding sign language motion data. Specifically, it includes a sign language dictionary database and a motion generation model.

[1973] "Means for rendering sign language motion data into a 3D animation format" refers to technology for rendering sign language motion data as 3D computer graphics to generate visually realistic sign language animation. Specifically, this refers to the use of 3D modeling software and rendering engines.

[1974] "Means for displaying the rendered animation to the user" refers to a video player component or display device that plays the generated 3D animation on the user's device, including a computer or mobile device display.

[1975] The present invention adds an emotion engine that recognizes the user's emotions to a system that translates sign language into text and a system that translates text into sign language. This improves the accuracy of sign language translation and realizes natural communication that reflects emotions. Specific embodiments of the present invention are described below.

[1976] Translating sign language into text

[1977] 1. Acquiring video data

[1978] The user signs.

[1979] The device (smartphone or camera-equipped device) records the sign language as a video, capturing the video in real time using the device's built-in camera.

[1980] The video data captured by the device is encoded in real time and sent to a server via the Internet using the RTMP protocol or WebRTC.

[1981] 2. Analysis of video data

[1982] The video data received by the server is divided into frames. This process uses a library such as FFmpeg.

[1983] The server analyzes each frame and uses OpenCV to identify the position and shape of the hand, using algorithms such as Edge Detection and Contour Detection.

[1984] The server uses a facial recognition algorithm (dlib, FaceNet, etc.) to extract the user's facial expressions from video frames, and then uses an emotion engine that utilizes natural language processing algorithms such as BERT to recognize the user's emotions.

[1985] 3. Sign Language Identification

[1986] The server inputs the analysis results into a sign language recognition model (a deep learning model built with TensorFlow or PyTorch), which analyzes the hand movements in each frame and identifies the corresponding sign language words and phrases.

[1987] The server combines the sign language recognition results with the emotions inferred by the emotion engine and adjusts the response.

[1988] 4. Conversion to Character Data

[1989] The server converts the sign language into a corresponding string of characters, choosing an appropriate expression that reflects the user's emotions. For example, for the sign "hello," the server converts it into "hello!", taking into account the smile.

[1990] The server encodes the generated character data and sends it to the terminal via the Internet.

[1991] The terminal decodes the received character data and displays it on the user's screen. Specifically, this is the operation of setting the character string in a GUI component.

[1992] Example (translating sign language into text)

[1993] The user smiles and signs "hello."

[1994] The device uses a camera to capture the sign language and facial expressions, and then sends the video to a server via the RTMP protocol.

[1995] The server splits the video into frames, analyzes hand shape and position using OpenCV, and analyzes facial expressions using dlib.

[1996] The server identifies "hello" using a TensorFlow model, and the emotion engine recognizes a smile using BERT.

[1997] The server generates the string "Hello!" and sends it to the terminal.

[1998] The terminal displays this string to the user.

[1999] Translating text into sign language

[2000] 1. Entering text data

[2001] The user enters text data (e.g., "Thank you") into the terminal, using a touchscreen or keyboard.

[2002] The device sends the input character data to the server via HTTP or WebSocket.

[2003] 2. Sign Language Generation

[2004] The server receives the character data and inputs it into a sign language generation algorithm (e.g., a deep learning model), which searches for corresponding sign language motion data from a character-to-sign language dictionary database.

[2005] The server uses an emotion engine to predict facial expressions from the input text and adjusts the sign language motion data based on that information. Specifically, it adjusts the movement and speed of the motion data to correspond to the emotion.

[2006] 3. Sending sign language animations

[2007] The server renders the generated sign language motion data using Blender or other 3D rendering software, and animates the 3D model of the sign language character.

[2008] The server encodes the 3D animation data and sends it to the device via the Internet.

[2009] 4. Displaying Sign Language Animation

[2010] The device decodes the received animation and plays it on the user's screen. Specifically, this is done by playing it using the video player component in the app.

[2011] The device displays emotional sign language motions to the user in 3D animation format.

[2012] Example (translating text into sign language)

[2013] The user types the word "thank you" into the device, for example, using the keyboard or touch screen of a smartphone.

[2014] The terminal sends the character data to the server via HTTP.

[2015] The server uses a sign language generation algorithm (TensorFlow model) to generate sign language motion data corresponding to "thank you."

[2016] The server uses an emotion engine (using the BERT model) to infer the emotion of gratitude from the input text and generates sign language motion that reflects that information.

[2017] The server uses Blender to render this motion data into a 3D animation, encodes it, and then sends it to the device.

[2018] The device decodes the animation and uses a video player to play the emotion-reflecting sign language animation on the user's screen.

[2019] Example prompts to input to the generative AI model

[2020] 1. Translating sign language to text: "When a user says 'hello' in sign language, analyze their hand movements and facial expressions and generate an appropriate string of text based on their emotions."

[2021] 2. Example of translating text into sign language: "Generate motion data to express the word 'thank you' in sign language, and create a 3D animation that reflects the emotion of gratitude."

[2022] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2023] Translating sign language into text

[2024] Step 1:

[2025] The user signs

[2026] Input: User's sign language actions

[2027] Specific action: The user signs to the camera, for example, signing "hello."

[2028] Output: Video data including the user's sign language

[2029] Step 2:

[2030] The device captures video and sends it to the server in real time.

[2031] Input: Video data containing the user's sign language

[2032] Specific operation: The system uses the device's camera to capture sign language video in real time, encodes the video data using the RTMP protocol, and then transmits the encoded video data to a server via the Internet.

[2033] Output: Video data sent to the server

[2034] Step 3:

[2035] The server divides the video data into frames and analyzes the shape and position of the hand.

[2036] Input: Video data

[2037] How it works: The server splits the video data into frames using the FFmpeg library. For each frame, it analyzes the hand shape and position using the Edge Detection and Contour Detection algorithms using OpenCV.

[2038] Output: Analysis results of hand shape and position

[2039] Step 4:

[2040] The server extracts facial expressions from video frames and recognizes emotions.

[2041] Input: Video data

[2042] Specific operation: The server uses dlib and FaceNet to detect facial features from frames, and uses BERT, a natural language processing algorithm, to recognize the user's emotions from the detected facial features.

[2043] Output: User emotion data

[2044] Step 5:

[2045] The server inputs the analysis results into a sign language recognition model and generates the corresponding string.

[2046] Input: Analysis results of hand shape and position, user emotion data

[2047] How it works: The server inputs the analysis results into a deep learning model built with TensorFlow or PyTorch, converts the sign language gestures into corresponding text, and uses the emotion data to select the appropriate expression method as needed.

[2048] Output: The corresponding string

[2049] Step 6:

[2050] The server sends the generated string to the terminal, which displays the string to the user.

[2051] Input: The corresponding string

[2052] Specific operation: The server encodes the generated string and sends it to the terminal via the Internet. The terminal decodes the received string and displays it in the GUI component.

[2053] Output: The string displayed on the user's screen

[2054] Translating text into sign language

[2055] Step 1:

[2056] The user enters character data

[2057] Input: Text data (e.g. "Thank you")

[2058] Specific Action: A user types text using the keyboard or touchscreen of a smartphone or tablet.

[2059] Output: Character data entered

[2060] Step 2:

[2061] The terminal sends the input character data to the server.

[2062] Input: Character data entered

[2063] Specific operation: The terminal sends character data to the server using HTTP or WebSocket.

[2064] Output: Character data sent to the server

[2065] Step 3:

[2066] The server converts the text data into sign language motion data.

[2067] Input: Character data

[2068] How it works: The server inputs character data into a sign language generation algorithm (e.g., a TensorFlow model) to generate sign language motion data. It then uses an emotion engine to infer emotions from the character data and adjusts the motion data based on the results.

[2069] Output: Sign language motion data

[2070] Step 4:

[2071] The server renders the sign language motion data into a 3D animation format.

[2072] Input: Sign language motion data

[2073] Specific operation: The server uses 3D rendering software such as Blender to render the sign language motion data into 3D animation.

[2074] Output: 3D animation data

[2075] Step 5:

[2076] The server sends 3D animation data to the terminal, and the terminal displays the animation to the user.

[2077] Input: 3D animation data

[2078] Specific operation: The server encodes 3D animation data and sends it to the device via the Internet. The device decodes the received animation data and plays it using the video player component.

[2079] Output: The animation played on the user's screen

[2080] (Application example 2)

[2081] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2082] Conventional sign language translation systems simply convert sign language into text, making it difficult to achieve natural communication that reflects the user's emotions. Furthermore, they lacked the ability to output text generated from sign language as speech, or the ability to convert text into sign language animation. This made it difficult for hearing-impaired people and people who use sign language to communicate smoothly on public transportation, such as self-driving vehicles.

[2083] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a video of a user's actions using sign language, means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame, means for inputting the analysis results into a sign language recognition model and generating a corresponding character string, means for extracting the user's facial expression from the video data and recognizing the emotion using an emotion engine, means for adjusting the character string to reflect the user's emotion, means for outputting the generated character string as audio, means for converting character data into sign language motion data, means for rendering the sign language motion data in a 3D animation format, and means for reflecting emotion in the rendered animation using the emotion engine. This enables users to communicate naturally and emotionally even inside an autonomous vehicle.

[2084] 1. "Means for capturing the actions of a user using sign language as video" refers to a device or system that records the actions of a user using sign language in video format using an image capture device such as a camera.

[2085] 2. "Means for dividing the captured video into frames and analyzing the shape and position of the hand in each frame" refers to an algorithm or software that breaks down video data into individual frames and analyzes the shape and position of the user's hand within each frame.

[2086] 3. "Means for inputting the analysis results into a sign language recognition model and generating a corresponding string of characters" refers to a process or system that inputs the analyzed hand movement data into a sign language recognition model and generates a corresponding string of characters as a result.

[2087] 4. "Means for displaying the generated character string to the user" refers to a device or system that outputs the generated character data to a display or other display device and presents it in a form that the user can visually confirm.

[2088] 5. "Means for extracting a user's facial expressions from video data and recognizing emotions using an emotion engine" means a process or system that extracts a user's facial expressions from video data and identifies the user's emotional state using machine learning or artificial intelligence algorithms.

[2089] 6. "Means for adjusting text to reflect user emotions" refers to an algorithm or process that appropriately changes the content and expression of generated text based on recognized user emotional information.

[2090] 7. "Means for audibly outputting the generated character string" refers to a device or system that uses speech synthesis technology to play back the generated character data as speech.

[2091] 8. "Means for converting character data into sign language motion data" means an algorithm or software that converts input character information into corresponding sign language motion data.

[2092] 9. "Means for rendering sign language motion data into a 3D animation format" means a process or system that renders the generated sign language motion data as a three-dimensional animation.

[2093] 10. "Means for reflecting emotions in rendered animation using an emotion engine" means a system or algorithm that applies an emotion engine to generated 3D animation and reflects emotions in the character's movements and expressions.

[2094] This invention relates to a system that enables users who use sign language to communicate naturally and emotionally within an autonomous vehicle. This system captures sign language movements as video and generates text, audio, and sign language animations based on the analysis results, thereby supporting comfortable communication for users.

[2095] System Program

[2096] The system uses the following hardware and software:

[2097] Hardware: In-vehicle camera, high-resolution display, on-board computer, audio output device

[2098] Software: Computer vision libraries (e.g., OpenCV), deep learning frameworks (e.g., TensorFlow, PyTorch), natural language processing libraries (e.g., NLTK, Transformers), emotion recognition engines

[2099] System action

[2100] 1. Acquiring video data:

[2101] The in-vehicle camera captures the user's sign language in real time as video, which is then broken down into frames and the shape and position of the hand in each frame are analyzed.

[2102] 2. Sign Language Analysis and Recognition:

[2103] The analyzed data is input into a sign language recognition model (deep learning algorithm) to generate a corresponding string of characters. At the same time, the user's facial expressions are extracted from the video data, and an emotion engine recognizes the user's emotions.

[2104] 3. Creating and displaying strings:

[2105] The system adjusts the text based on the recognized sign language and emotional information, and the resulting text is displayed on the vehicle's display. The text is also output as speech using speech synthesis technology.

[2106] 4. Generating Sign Language Animation:

[2107] Conversely, text data entered by the user is converted into sign language motion data, which is then rendered into a 3D animation format and emotion-based using an emotion recognition engine. The final rendered animation is then displayed on the screen.

[2108] Specific examples

[2109] Translating sign language into text and speech

[2110] 1. A user signs "thank you" inside a self-driving vehicle.

[2111] 2. Cameras inside the vehicle capture sign language in real time and transmit the data to an onboard computer.

[2112] 3. The on-board computer divides the video into frames and analyzes hand movements and the user's facial expressions.

[2113] 4. Based on the analysis results, the sign language recognition model and emotion engine generate the string "Thank you" and emotional information.

[2114] 5. "Thank you" will appear on the display and be output as audio at the same time.

[2115] Translating text into sign language animation

[2116] 1. The user types "Help me" into the display.

[2117] 2. The on-board computer converts the text data into sign language motion data, and the emotion engine recognizes and reflects the emotion of "urgency."

[2118] 3. The generated sign language motion data is rendered as a 3D animation and displayed on the screen.

[2119] Prompt Sentence Examples

[2120] Translate sign language to text: A user signs "Where is my house?" You can see confusion in the user's face.

[2121] Information from sign language:

[2122] Transcription: Where is my house?

[2123] Emotion: Confused

[2124] Translate text into sign language: The user has typed "Thank you." Reflect the sentiment of gratitude with the emotion engine.

[2125] Generated sign language motion:

[2126] Sign Language: Thank you

[2127] Emotion: Gratitude

[2128] This will enable users to have natural and emotionally rich communication even inside an autonomous vehicle.

[2129] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2130] Step 1:

[2131] Acquiring video data using a camera: When a user signs, the device's in-car camera captures the action in real time. This captures video data including the user's hand movements and facial expressions. The input is "video data including the user's sign language and facial expressions," and the output is "video data acquired in real time."

[2132] Step 2:

[2133] Video data frame division and analysis: The server receives the acquired video data and divides the video into frames. The shape and position of the hand in each frame are analyzed using a computer vision library (e.g., OpenCV). The input is "video data acquired in real time," and the output is "hand shape and position data for each frame."

[2134] Step 3:

[2135] Sign language recognition: The server inputs the analyzed frame data into a sign language recognition model (e.g., a deep learning algorithm using TensorFlow or PyTorch) to identify the corresponding sign language words or phrases. The input is the hand shape and position data for each frame, and the output is the identified sign language string.

[2136] Step 4:

[2137] Facial expression analysis and emotion recognition: The server extracts the user's facial expressions from the video data and recognizes the user's emotions using a natural language processing algorithm and an emotion engine. The input is "video data acquired in real time" and the output is "user's emotional information."

[2138] Step 5:

[2139] Generating and displaying text and audio: The server adjusts the text based on the recognized sign language text and emotional information, and displays it on the vehicle's display. The text is also output as audio using a speech synthesis engine. The input is the "identified sign language text and the user's emotional information," and the output is "text and audio data displayed to the user."

[2140] Step 6:

[2141] Conversion of character data into sign language motion data: The server receives character data entered by the user and converts it into corresponding sign language motion data. The input is "character data entered by the user" and the output is "sign language motion data."

[2142] Step 7:

[2143] Rendering sign language motion data and reflecting emotions: The server renders the sign language motion data into 3D animation format and reflects emotional information using an emotion engine. The input is "sign language motion data" and "emotional information," and the output is "emotionally reflected sign language 3D animation."

[2144] Step 8:

[2145] Displaying sign language animation: The device displays the rendered sign language 3D animation on the display, providing a visual experience to the user. The input is "emotion-reflecting sign language 3D animation," and the output is "sign language 3D animation displayed to the user."

[2146] Through these steps, users can communicate naturally and emotively using sign language inside a self-driving vehicle.

[2147] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2148] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2149] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2150] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2151] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2152] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2153] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2154] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2155] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2156] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2157] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2158] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2159] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2160] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2161] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2162] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2163] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2164] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2165] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2166] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2167] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2168] The following is further disclosed regarding the above embodiment.

[2169] We propose the following scope of claims:

[2170] (Claim 1)

[2171] means for acquiring a video of a user's actions while using sign language;

[2172] A means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame;

[2173] a means for inputting the analysis results into a sign language recognition model to generate a corresponding string of characters;

[2174] means for displaying the generated string to a user;

[2175] A system including:

[2176] (Claim 2)

[2177] 10. The system of claim 1, wherein the sign language recognition model utilizes a deep learning algorithm.

[2178] (Claim 3)

[2179] a means for converting character data into sign language motion data;

[2180] a means for rendering sign language motion data into a 3D animation format;

[2181] means for displaying the rendered animation to a user;

[2182] The system of claim 1 further comprising:

[2183] "Example 1"

[2184] (Claim 1)

[2185] A means for capturing a user's actions as a video;

[2186] A means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame;

[2187] a means for inputting the analysis results into a sign language recognition model and converting the sign language actions into corresponding character strings;

[2188] means for displaying the generated string to the user in real time;

[2189] A system including:

[2190] (Claim 2)

[2191] 10. The system of claim 1, wherein the sign language recognition model utilizes a deep learning algorithm.

[2192] (Claim 3)

[2193] a means for converting character data into sign language motion data;

[2194] a means for rendering sign language motion data into a 3D animation format;

[2195] means for displaying the rendered animation to a user;

[2196] The system of claim 1 further comprising:

[2197] "Application Example 1"

[2198] (Claim 1)

[2199] means for acquiring a video of a user's actions while using sign language;

[2200] A means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame;

[2201] a means for inputting the analysis results into a sign language recognition model to generate a corresponding string of characters;

[2202] means for presenting the generated character string on a displayable device; ...

Claims

1. means for acquiring a video of a user's actions while using sign language; A means for dividing the acquired video into frames and analyzing the shape and position of the hand in each frame; a means for inputting the analysis results into a sign language recognition model to generate a corresponding string of characters; means for displaying the generated string to a user; A system including:

2. 10. The system of claim 1, wherein the sign language recognition model utilizes a deep learning algorithm.

3. a means for converting character data into sign language motion data; a means for rendering sign language motion data into a 3D animation format; means for displaying the rendered animation to a user; The system of claim 1 further comprising:

Images