system

Abstract

Provide a system. A method for receiving voice or text input; a means for converting input speech into text; A means of analyzing text and understanding emotions and nuances, a means for generating a sign language video based on the analysis result; a means for displaying the generated sign language video; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Unlike other languages, sign language is a means of communication that is difficult for hearing people to learn voluntarily, making communication with the hearing impaired generally difficult. There are also issues such as a shortage of sign language interpreters and the difficulty of appropriately expressing subtle nuances of language and emotions in sign language. This creates the problem of inefficient and smooth communication that incorporates emotions and nuances between hearing impaired and hearing people. [Means for solving the problem]

[0005] The present invention provides a system including means for accepting voice or text input, means for converting the input voice into text, means for analyzing the text to understand emotions and nuances, means for generating a sign language video based on the analysis results, and means for displaying the generated sign language video.The system also includes means for capturing a user's sign language actions using a camera, means for analyzing the captured sign language actions and converting them into corresponding text, and means for converting the converted text into voice.The system further includes means for reflecting the strength and nuance of emotions in the sign language video based on the context and emotions of the analyzed text, thereby enabling communication that accurately conveys emotions and nuances and enabling smooth two-way communication between hearing-impaired people and those with hearing impairments.

[0006] "Voice" is a form of information, including human spoken words and sounds, that is converted into digital data through a device such as a microphone.

[0007] "Text" is a form of digital data that expresses information using characters and symbols, and is generated by character input devices, text processing software, etc.

[0008] "Means for accepting input" refers to an interface or device for acquiring voice or text data from a user and processing it within the system.

[0009] "Means for converting voice to text" refers to technology or devices that use voice recognition technology to convert input voice data into corresponding text data.

[0010] "Means for analyzing text" refers to technologies and devices that use natural language processing technology to analyze input text data and understand its context, sentiment, and nuances.

[0011] "Means for understanding emotions and nuances" refers to technologies and devices for extracting and understanding a user's emotional state and contextual nuances from the results of text analysis.

[0012] "Means for generating sign language video" refers to technology or equipment that generates appropriate sign language movements in video format based on the analysis results.

[0013] "Means for displaying sign language video" refers to a display device or playback software for visually displaying the generated sign language video.

[0014] "Means for capturing sign language actions using a camera" refers to techniques or devices that record the sign language actions of a user using a device such as a video camera.

[0015] "Means for analyzing sign language actions" refers to technology or devices that analyze captured sign language actions and convert them into corresponding text data.

[0016] "Text-to-speech conversion means" refers to a technology or device that converts input text data into corresponding voice data using voice synthesis technology.

[0017] "Means for reflecting emotional intensity and nuances" refers to technologies and devices that reflect the analyzed emotions and contextual nuances in the sign language video that is generated. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention provides a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. As an embodiment of the present invention, the process of the program is described below in stages.

[0040] Roles of various devices

[0041] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and conversion processing.

[0042] User input

[0043] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the input is done using a keyboard or touch screen.

[0044] Speech recognition and text capture

[0045] For voice input, the device uses a speech recognition engine (e.g., Google® Speech-to-Text API) to convert the voice into text data. For text input, the device uses the text data entered by the user as is.

[0046] Text analysis and nuance understanding

[0047] The device sends the text data to a server for processing. The server uses natural language processing (NLP) techniques to analyze the text data and understand the context, sentiment, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[0048] Sign language video generation

[0049] The server generates appropriate sign language animations based on the analyzed data. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the strength of gratitude.

[0050] Sign language video distribution to users

[0051] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0052] Sign language to text and speech translation

[0053] When a user communicates information using sign language, the device's camera captures the sign, which the device then analyzes using a sign language recognition engine and converts into corresponding text, which it then converts into speech using a speech synthesis engine and plays back to the user.

[0054] Specific example explanation

[0055] For example, if User A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures this sign language action and the server converts it into the corresponding text "You're welcome." The device converts this text into speech and plays it back to User A.

[0056] In this way, the system according to the present invention realizes highly accurate conversion of voice and text input into sign language video, and from sign language to text and voice, supporting smooth communication between users.

[0057] The processing flow will be explained below.

[0058] Step 1:

[0059] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[0060] Step 2:

[0061] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[0062] Step 3:

[0063] The device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert the speech into the text "Thank you." If text is entered, the entered text "Thank you" is used as is.

[0064] Step 4:

[0065] The terminal transmits the acquired text "Thank you" to the server.

[0066] Step 5:

[0067] The server receives the text "Thank you" sent from the terminal.

[0068] Step 6:

[0069] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[0070] Step 7:

[0071] Based on the analysis results, the server generates appropriate sign language video data, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[0072] Step 8:

[0073] The server transmits the generated sign language video data to the terminal.

[0074] Step 9:

[0075] The terminal acquires the sign language video data received from the server.

[0076] Step 10:

[0077] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone.

[0078] Step 11:

[0079] The user signs "hello" to the camera.

[0080] Step 12:

[0081] The device uses a camera to capture the user's sign language actions.

[0082] Step 13:

[0083] The device analyzes the captured sign language gestures using a sign language recognition engine and converts them into the corresponding text "Hello."

[0084] Step 14:

[0085] The device uses a speech synthesis engine (e.g., Text-to-Speech API) to convert the converted text into the voice "Hello" and plays it back to the user.

[0086] In this way, the system can smoothly convert voice and text input into sign language video and sign language back into text and voice, enabling real-time communication between users that includes emotions and nuances.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Conventional systems that convert speech or text into sign language may not generate sign language videos accurately. Furthermore, the conversion from sign language to text and speech is also inaccurate, often resulting in poor communication between users. Furthermore, there is a lack of technology that can understand the context and emotions of text and appropriately reflect them in sign language videos.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes: means for accepting voice or text input; means for converting the input voice into text; means for analyzing the text and understanding emotions and nuances; means for generating a sign language video based on the analysis results; means for transmitting the generated sign language video to a terminal; means for playing the generated sign language video and visually displaying it to a user; means for capturing a user's sign language movements using a camera; means for analyzing the captured sign language movements and converting them into corresponding text; means for converting the converted text into voice; and means for reflecting the strength and nuance of emotions in the sign language video based on the context and emotions of the analyzed text. This enables highly accurate conversion from voice or text to sign language video and from sign language to text and voice, enabling smooth communication between users.

[0092] "Means for accepting voice or text input" is a general term for devices and software that recognize voice spoken or text entered by a user and incorporate it into the system.

[0093] "Means for converting input speech into text" refers to software or a service that uses a speech recognition engine to analyze the speech uttered by the user and convert it into corresponding text data.

[0094] "Means for analyzing text and understanding emotions and nuances" refers to algorithms or software that use natural language processing technology to analyze text data and extract and understand its context, emotions, and nuances.

[0095] The "means for generating sign language video based on the analysis results" refers to a system that includes a database and video generation technology for generating appropriate sign language video based on the analyzed text data and its emotions and nuances.

[0096] The "means for transmitting the generated sign language video to the terminal" refers to a technology including a communication means and protocol for transferring the generated sign language video data to the user's terminal.

[0097] "Means for playing the generated sign language video and visually displaying it to the user" refers to a display and playback control software that plays the sign language video on the terminal and allows the user to visually confirm the video.

[0098] A "means for capturing a user's sign language actions using a camera" is a device that includes a camera and its control software for recording a user's sign language actions in real time.

[0099] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to a sign language recognition algorithm or software that analyzes video footage of captured sign language actions and generates text data corresponding to those actions.

[0100] "Converted text to speech conversion means" refers to a speech synthesis engine and associated software for converting text data into speech data and playing it back to the user.

[0101] "Means for reflecting emotional strength and nuance in sign language video based on the context and emotion of analyzed text" is a technology for adding appropriate emotional expressions and nuance to sign language video using the context and emotion information of analyzed text data.

[0102] The present invention provides a system for converting voice and text input into sign language video in real time, and then converting sign language into text and voice. The configuration and specific implementation of this system are described in detail below.

[0103] System configuration

[0104] This system mainly consists of three main elements: users, terminals, and servers.

[0105] User operations

[0106] The user inputs voice or text into the terminal using an input device such as a smartphone or microphone. For example, the user may say "Good morning."

[0107] Users can also communicate information using sign language, which is captured by the device's camera.

[0108] Device Features

[0109] The device captures voice data from the user and converts the voice into text data using a speech recognition engine such as the Google Speech-to-Text API. In the case of text input, the device uses the text data entered by the user as is.

[0110] The terminal transmits the acquired text data to the server.

[0111] The device has the ability to receive and play sign language videos sent from the server.

[0112] The device captures the user's sign language input with a camera, analyzes the data with a sign language recognition engine, converts it into text, and then converts the converted text into speech using a speech synthesis engine, which plays it back to the user.

[0113] Server Features

[0114] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) technology, which allows it to understand the context, sentiment, and nuances.

[0115] The server generates appropriate sign language animations based on the analysis results, using a database of sign language actions to generate new sign language animations as needed.

[0116] The server transmits the generated sign language video to the terminal.

[0117] Specific example explanation

[0118] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much" using the Google Speech-to-Text API. The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and converts it into the text "You're welcome" using a sign language recognition engine. The device then converts this text into speech using a speech synthesis engine and plays it back to user A. This process enables smooth communication between users.

[0119] Prompt Sentence Examples

[0120] If the user types "hello" via voice or text, explain how you would process this input and convert it into a sign language video, and then from sign language back to text and speech.

[0121] This system converts voice and text input into sign language video, and sign language into text and voice with high accuracy, supporting smooth communication between users.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1: User types into terminal

[0124] The user inputs speech or text into the terminal.

[0125] Input: User's voice "Good morning" or text "Good morning".

[0126] Action: A user says "Good morning" to their smartphone or types "Good morning" on their keyboard.

[0127] Output: The captured audio or text data.

[0128] Step 2: Speech recognition or text capture

[0129] The device receives voice data from the user and converts it into text data using the Google Speech-to-Text API.

[0130] Input: Audio data.

[0131] How it works: The device uses a speech recognition engine to convert speech into text, for example, converting "Good morning" into text data.

[0132] Output: Text data "Good morning".

[0133] Step 3: Sending text data to the server

[0134] The terminal transmits the acquired text data to the server.

[0135] Input: Text data "Good morning".

[0136] Operation: The device sends text data to the server over the network.

[0137] Output: Text data passed to the server.

[0138] Step 4: Text Analysis with NLP

[0139] The server analyzes the received text data to understand the context, sentiment, and nuances.

[0140] Input: Received text data "Good morning."

[0141] How it works: The server uses natural language processing techniques to analyze the text. For example, "Good morning" is recognized as a common greeting.

[0142] Output: Analyzed contextual information and sentiment data.

[0143] Step 5: Generate sign language video

[0144] The server generates appropriate sign language video based on the analyzed data.

[0145] Input: Analyzed contextual information and sentiment data.

[0146] Action: The server refers to a database of sign language actions and generates a sign language animation that expresses "good morning."

[0147] Output: The generated sign language video.

[0148] Step 6: Streaming sign language videos to devices

[0149] The server sends the generated sign language video to the terminal.

[0150] Input: Generated sign language video.

[0151] Operation: The server sends video data to the device via the network.

[0152] Output: Sign language video received on the device.

[0153] Step 7: Play and view the sign language video

[0154] The terminal plays the received sign language video and displays it visually to the user.

[0155] Input: Received sign language video data.

[0156] Operation: The device plays the received sign language video and displays it on the display. For example, a sign language video of "Good morning" is displayed to User B.

[0157] Output: Sign language video displayed to the user.

[0158] Step 8: Translate sign language to text and speech

[0159] The user communicates information in sign language.

[0160] Input: The user's sign language gesture.

[0161] How it works: The camera captures sign language gestures, which the device analyzes using a sign language recognition engine and converts into corresponding text, which is then converted into speech using a speech synthesis engine.

[0162] Output: Converted text and audio data. When User A replies "Good morning" in sign language, the text and audio "Good morning" are played back to User A.

[0163] In this way, this system achieves smooth communication between users through each step.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] Conventional systems for converting speech or text into sign language have had the problem of difficulty in real-time communication between different language modes. Furthermore, converting sign language into text or speech is complex, making smooth communication difficult, especially in face-to-face environments such as brick-and-mortar stores. This has resulted in a lack of support for customers and store staff with disabilities to communicate efficiently.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes means for accepting voice or text input, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, and means for playing back the generated sign language video, thereby enabling efficient real-time communication between customers with disabilities and store staff in physical stores.

[0169] "Voice" refers to the ability to electronically record human voices and process them as digital data.

[0170] "Characters" are a collection of symbols that electronically express a person's intentions or information.

[0171] "Means for accepting input" refers to a device or function that receives voice or text from a user and converts that data into a format that can be processed within the system.

[0172] A "speech-to-text means" is a technology that analyzes received speech data and converts it into a corresponding string of written symbols.

[0173] "A means of analyzing characters and understanding emotions and nuances" is a technology that analyzes the usage and structure of words in a sentence and extracts the emotions and subtle meanings behind them.

[0174] "Means for generating sign language images" refers to a technology that uses analyzed text information to create visual data that mimics human hand and body movements and is expressed as sign language.

[0175] "Means for playing the generated sign language video" refers to a technology for visually presenting the generated sign language video to a user using an appropriate display device.

[0176] "Video equipment" refers to cameras and related devices that can capture user movements and process the data in real time.

[0177] "Sign gestures" refer to specific movements and positions of the hands and body for the purpose of communication, thereby creating a visual representation of language.

[0178] "Means for capturing sign language actions" refers to technology that uses a device such as a camera to record a user's sign language actions as digital data.

[0179] The "means for analyzing and converting into corresponding characters" is a technology for analyzing the data of the captured sign language movements and generating a written symbol string corresponding to the movements.

[0180] A "means of converting text into speech" is a technology that analyzes a written string of symbols and expresses that string of symbols in speech.

[0181] "Context" refers to the context of the text and the circumstances surrounding its use, and is an element that forms the meaning of the text as a whole.

[0182] "Emotions" refers to a person's internal sensations and feelings, and focuses on how they affect communication.

[0183] A system for implementing this invention includes a device that accepts voice or text input, a function that converts the input voice into text, a function that analyzes the text and understands emotions and nuances, a function that generates sign language video based on the analysis results, and a function that plays back the generated sign language video.

[0184] The server is responsible for this complex data processing and conversion. Specifically, it uses the Google Speech-to-Text API to acquire audio data. It also uses natural language processing (NLP) technology to analyze text data, using deep learning frameworks such as Tensorflow (registered trademark). It uses the some_sign_language_video_generation_module, which is based on a database of sign language actions, to generate sign language videos.

[0185] The terminal is a device that accepts input from the user and displays the processing results. Voice input is done from a microphone or smartphone, and text input is done from a keyboard or touch screen. Sign language images are displayed using "OpenCV."

[0186] The camera is used to capture sign language movements, which are then converted into text using some_sign_language_recognition_module, which is then converted back into audio using the Google Text-to-Speech API.

[0187] As a concrete example, consider the case where a user asks "Where is this product?" in a physical store. The device captures the voice and converts it into text "Where is this product?" The server analyzes the text to understand the emotion and context and generates a sign language video. The generated sign language video is sent to the device and played back to the user.

[0188] When a store staff member replies in sign language, "This way," the camera captures this sign and converts it into text. The converted text, "This way," is then converted into speech by the server and played back to the user. This process enables efficient real-time communication between customers with disabilities and store staff in physical stores.

[0189] A specific example of a prompt for a generative AI model is, "A customer asks by voice, 'Where is this product?' Please design and implement a system that includes voice capture, text conversion, sign language video generation, sign language recognition, and speech synthesis." This prompt makes the overall system flow easier to understand.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The device accepts voice input from the user. When the user says, "Where is this product?", the device's microphone captures the voice. The captured voice data becomes input data. This allows the voice data to be collected and processed in the next step.

[0193] Step 2:

[0194] The device converts the captured voice data into text data. The data is sent to the server, where it is analyzed using the Google Speech-to-Text API, resulting in the text data "Where is this product?" This is the process of speech recognition and speech-to-text conversion.

[0195] Step 3:

[0196] The server analyzes the text data it receives to understand emotions and nuances. It uses natural language processing technology to analyze the context of the sentence and extract the intention and emotion behind the question, "Where is this product?". TensorFlow and other tools are used to analyze the data. As a result, the text and its contextual information are output.

[0197] Step 4:

[0198] The server generates a sign language video based on the analysis results. Using the sign language action database and some_sign_language_video_generation_module, it generates a sign language video corresponding to "Where is this product?" The generated sign language video becomes the output data.

[0199] Step 5:

[0200] The generated sign language video is sent to a terminal for playback. The terminal receives the sign language video and plays it back on a display device. The user can understand the intended question from the displayed sign language video.

[0201] Step 6:

[0202] The user (store staff member) responds in sign language, saying "This way." The camera captures this sign language action in real time. The captured sign language action becomes input data. This records the sign language response as digital data.

[0203] Step 7:

[0204] The server analyzes the sign language movements and converts them into corresponding text data. Using some_sign_language_recognition_module, the captured sign language movements are analyzed and converted into text data such as "This way." This is the step where the sign language movements are extracted as text information.

[0205] Step 8:

[0206] The server converts the text data into audio and plays it back to the user (customer). Using the Google Text-to-Speech API, the text data "This way" is converted into audio data and sent to the device. The device outputs this audio data using a playback device, allowing the user (customer) to receive an audio response.

[0207] This series of processes enables customers with disabilities and store staff to communicate efficiently in real time in physical stores.

[0208] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0209] The present invention is a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. The following describes the program process in stages as an embodiment of the present invention.

[0210] Roles of various devices

[0211] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and data conversion processing.

[0212] User input

[0213] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the user inputs the text using a keyboard or touch screen. Furthermore, by using an emotion engine, the user can include emotions when inputting.

[0214] Speech recognition and text capture

[0215] For voice input, the device uses a voice recognition engine to convert the voice into text data. For text input, the device uses the text data entered by the user as is. The emotion engine recognizes emotions from the user's voice and text, and includes the results in the transmitted data.

[0216] Text analysis and nuance understanding

[0217] The device sends the recognized emotion information along with the text data to a server, which then uses natural language processing (NLP) technology to analyze the text data and understand the context, emotion, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[0218] Sign language video generation

[0219] The server generates appropriate sign language animations based on the analyzed text and emotional information. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the intensity of gratitude.

[0220] Sign language video distribution to users

[0221] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0222] Sign language to text and speech translation

[0223] When a user communicates information using sign language, the device's camera captures the sign. The device analyzes the captured sign using a sign language recognition engine and converts it into corresponding text. The text is then converted into speech using a speech synthesis engine and played back to the user. An emotion engine also operates to extract additional emotional information from the user's facial expressions and actions and reflect it in the speech and text.

[0224] Specific example explanation

[0225] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The emotion engine then detects user A's emotion and determines it to be "strong gratitude." The server analyzes this text and emotion and generates a sign language video expressing strong gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and the server converts it into the corresponding text, "You're welcome." The device converts this text into speech and plays it back as "You're welcome." At the same time, the emotion engine extracts emotion from user B's sign language movements and facial expressions and reflects it in the text and speech.

[0226] In this way, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[0227] The processing flow will be explained below.

[0228] Step 1:

[0229] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[0230] Step 2:

[0231] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[0232] Step 3:

[0233] The device uses a speech recognition engine to convert the speech into the text "Thank you." In the case of text input, the text "Thank you" entered by the user is used as is.

[0234] Step 4:

[0235] The device uses an emotion engine to recognize emotions from the user's voice and text, along with the acquired text "Thank you." The emotion engine extracts emotions from the user's voice intonation and text expression, and transmits the emotion information along with the text data to the server.

[0236] Step 5:

[0237] The server receives the text "Thank you" and the emotion information sent from the terminal.

[0238] Step 6:

[0239] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[0240] Step 7:

[0241] The server generates appropriate sign language video data based on the analysis results, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[0242] Step 8:

[0243] The server transmits the generated sign language video data to the terminal.

[0244] Step 9:

[0245] The terminal acquires the sign language video data received from the server.

[0246] Step 10:

[0247] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone. This sign language video reflects the strength and nuance of the gratitude.

[0248] Step 11:

[0249] The user signs "hello" to the camera.

[0250] Step 12:

[0251] The terminal uses a camera to capture the user's sign language actions.

[0252] Step 13:

[0253] The device analyzes the captured sign language gestures with a sign language recognition engine and converts them into the corresponding text "Hello." At the same time, the emotion engine extracts emotions from the user's facial expressions and sign language gestures and includes the emotion information.

[0254] Step 14:

[0255] The terminal transmits the converted text and emotion information to the server.

[0256] Step 15:

[0257] The server analyzes the received text "Hello" and the emotional information, and generates a corresponding voice. The tone and intensity of the voice are adjusted based on the emotional information.

[0258] Step 16:

[0259] The server transmits the generated voice data to the terminal.

[0260] Step 17:

[0261] The terminal plays back the audio data received from the server. By listening to this audio, the user can understand the meaning and emotion of "hello" expressed in sign language.

[0262] This specific procedure allows the system to smoothly convert voice or text input into sign language video, and sign language into text and voice. Furthermore, an emotion engine is used to enable communication that takes into account the user's emotions, enabling real-time two-way communication that includes emotions and nuances.

[0263] Example 2

[0264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] Conventional communication systems have had issues with not being able to convert speech / text to sign language and vice versa in sufficient real time, resulting in inaccurate transmission of emotions and nuances. Furthermore, when generating sign language videos while taking the user's emotions into account, or converting sign language to text and speech, emotional information is not reflected, making it difficult to achieve natural and rich communication.

[0266] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0267] In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, means for displaying the generated sign language video, means for recognizing the user's emotions and generating the emotions as data, and means for transmitting the emotion data to the server and combining it with the analysis results, thereby enabling real-time communication that includes emotions and nuances.

[0268] A "means for accepting voice or text input" is a device or program that receives voice input and text input from a user and stores them as data.

[0269] A "means for converting input speech to text" is a device or program that uses a speech recognition engine to analyze a speech signal and convert it into a corresponding string of characters.

[0270] "Means for analyzing text and understanding emotions and nuances" refers to a device or program that uses natural language processing technology to analyze the context and emotions of text data and understand its content.

[0271] The "means for generating sign language video based on the analysis results" refers to a device or program that creates appropriate sign language video based on the analyzed text and emotional information.

[0272] The "means for displaying the generated sign language video" refers to a display device or playback application that visually presents the generated sign language video to the user.

[0273] The "means for recognizing a user's emotions and generating those emotions as data" refers to a device or program that uses an emotion recognition engine that analyzes emotions from the user's voice or text and converts them into data.

[0274] The "means for transmitting emotion data to a server and combining it with the analysis results" refers to a device or program that transmits emotion data generated by an emotion recognition engine to a server and integrates it with the text analysis results on the server side.

[0275] The "means for capturing a user's sign language actions using a camera" refers to a device that uses a video camera to capture a user's sign language actions and record them as digital data.

[0276] A "means for analyzing captured sign language actions and converting them into corresponding text" is a device or program that uses a sign language recognition engine to analyze captured sign language actions and generate corresponding text.

[0277] The "converted text to speech conversion means" is a device or program that uses a speech synthesis engine to convert text data into speech data.

[0278] "Means for extracting emotional information from a user's facial expressions and movements and generating it as data" refers to a device or program that uses an emotion recognition engine to analyze a user's facial expressions and movements recognized by a camera or the like and convert those emotions into data.

[0279] The "means for reflecting the generated emotional data in text or voice" refers to a device or program that adjusts the nuances of the text or voice based on the emotional data.

[0280] "Means for referencing the database necessary for generating sign language animation and generating new sign language animation as needed" refers to a device or program that uses a database of sign language actions to generate new or customized sign language animation.

[0281] "Means for transmitting the generated sign language video to a terminal and playing it" refers to a device or program that transmits the sign language video generated by the server to a user's terminal and plays the video on that terminal.

[0282] The present invention is a system that converts voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. Specific embodiments of this system are described in detail below.

[0283] User input

[0284] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "Thank you" by voice, the device captures this voice data using a smartphone or microphone. If the user inputs "Thank you" by text, the device uses a keyboard or touchscreen.

[0285] Speech recognition and text capture

[0286] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text. For text input, the device uses the text entered by the user as is.

[0287] emotion recognition

[0288] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[0289] Data transmission

[0290] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[0291] Text analysis and nuance understanding

[0292] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 (registered trademark) is used for this purpose.

[0293] Sign language video generation

[0294] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[0295] Sign language video distribution to users

[0296] The sign language video generated by the server is sent to the terminal, which plays the sign language video and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0297] Sign language to text and speech translation

[0298] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS® Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[0299] Specific example explanation

[0300] For example, if User A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects User A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from User B's sign language movements and facial expressions and reflects them in the text and speech.

[0301] Prompt Sentence Examples

[0302] An example of a prompt you can use is:

[0303] "Describe a system that converts user-supplied text into sign language video. It includes an emotion engine and understands nuance."

[0304] As a result, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[0305] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0306] Step 1:

[0307] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "thank you" by voice, the device captures this voice data using a smartphone or microphone.

[0308] Input: User speech, text, sign language

[0309] Output: Captured voice data, text data, sign language data

[0310] Step 2:

[0311] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text data.

[0312] Input: Captured audio data

[0313] Output: Text data generated by speech recognition

[0314] Step 3:

[0315] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[0316] Input: Text data, audio data

[0317] Output: Emotion data

[0318] Step 4:

[0319] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[0320] Input: Text data, emotion data

[0321] Output: Data sent to the server

[0322] Step 5:

[0323] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[0324] Input: Received text data, emotion data

[0325] Output: Parsed text data and sentiment data

[0326] Step 6:

[0327] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[0328] Input: Parsed text data and emotion data

[0329] Output: Generated sign language video

[0330] Step 7:

[0331] The server sends the generated sign language video to the device, which then plays it back and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0332] Input: Generated sign language video

[0333] Output: Sign language video sent to the device, user receives visual information

[0334] Step 8:

[0335] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[0336] Input: Captured sign language gestures

[0337] Output: Text data, audio data

[0338] Specifically, for example, if user A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects user A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from user B's sign language movements and facial expressions and reflects them in text and speech.

[0339] (Application example 2)

[0340] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0341] In conventional security services, it has been difficult for hearing-impaired people to quickly and accurately report emergencies and receive appropriate assistance. Furthermore, there is a lack of a means to accurately convey emotions and nuances of urgency when communicating using sign language. This can lead to delays in emergency communications and misunderstandings.

[0342] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating sign language video based on the analysis results, means for displaying the generated sign language video, means for analyzing the user's emotions in an emergency and generating appropriate sign language video, and means for analyzing the user's sign language movements in real time and converting them into voice or text. This enables hearing-impaired people to report emergencies quickly and accurately and receive appropriate support. Furthermore, nuances of emotions and urgency can be accurately conveyed in communication using sign language.

[0343] "Means for accepting voice or text input" refers to a device or interface that a user can use to input voice or text.

[0344] "Means for converting input speech to text" refers to an algorithm or engine that uses speech recognition technology to convert speech data into corresponding text data.

[0345] "Means of analyzing text and understanding emotions and nuances" refers to the function of analyzing text data using natural language processing technology and extracting context, emotions, and nuances.

[0346] "Means for generating sign language videos based on the analysis results" refers to the function for generating appropriate sign language actions based on the analyzed text and emotional information and outputting them in video format.

[0347] "Means for displaying the generated sign language video" refers to hardware and software such as a monitor or display for visually displaying the generated sign language video to the user.

[0348] "Means for analyzing a user's emotions in an emergency and generating appropriate sign language video" refers to a function that quickly analyzes a user's emotions in an emergency and generates emphasized sign language video based on that emotional information.

[0349] "Means for analyzing a user's sign language actions in real time and converting them into speech or text" refers to technologies or algorithms that analyze and convert sign language actions captured using a camera into corresponding speech or text in real time.

[0350] "Means for capturing a user's sign movements using a camera" refers to a camera or related device used to capture the sign movements.

[0351] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to an algorithm or engine for analyzing captured sign language actions and generating text corresponding to those actions.

[0352] "Means for converting converted text into speech" refers to a function for converting text data into speech data using speech synthesis technology.

[0353] "Means for reflecting a user's emotional information in an emergency situation in voice and text" refers to a technology that appropriately reflects emotional information obtained from a user in an emergency situation in the generated voice and text.

[0354] "Means for reflecting emotional strength and nuance in sign language videos based on the context and emotion of the analyzed text" refers to a function that adds emotional strength and nuance to the actions of sign language videos based on the context and emotion information of the analyzed text.

[0355] "Means for adding special nuances to sign language videos through emotion analysis in emergency situations" refers to technology that analyzes particularly heightened emotions in emergency situations and reflects the results in sign language videos to add special nuances.

[0356] This invention is a speech-to-sign language conversion system specialized for security services, which helps hearing-impaired people to communicate quickly and accurately even in emergency situations. The configuration and operation of this system are described in detail below.

[0357] Overall system configuration

[0358] This system consists of a terminal that accepts user input, a server that analyzes the data and generates sign language video, and a device that provides the generated sign language video or audio to the user.

[0359] Device Features

[0360] The user reports an emergency by voice or text. The device is equipped with a voice input device (e.g., microphone) and a text input interface (e.g., keyboard, touchscreen). If the user inputs by voice, the device uses a speech recognition engine (e.g., Google Speech Recognition) to convert the voice into text. Along with the converted text, an emotion engine analyzes the user's emotions and sends that information to the server.

[0361] Server Features

[0362] The server uses natural language processing (NLP) technology to analyze the context and nuances of the text based on the received text and emotional information. For example, for a highly urgent text such as "Help me," it generates a sign language video with a strong emotional expression. A dedicated database and sign language motion generation technology are used to generate the sign language video.

[0363] Displaying the generated sign language video

[0364] The sign language video generated by the server is sent to the device and displayed visually to the user, allowing hearing-impaired people to quickly report their situation and receive appropriate assistance.

[0365] Sign language to text and speech translation

[0366] When a user responds in sign language, the camera on the device captures the sign. The server analyzes the captured sign and converts it into corresponding text. This text is then converted into speech and played back to the user. The sign language recognition engine and speech synthesis engine used are similarly sophisticated.

[0367] The specific hardware and software used

[0368] Audio input device: Microphone

[0369] Text input interface: keyboard, touchscreen

[0370] Speech recognition engine: Google Speech Recognition

[0371] Natural language processing technology: NLP engine

[0372] Emotion Engine: Emotion Recognition

[0373] Sign language motion generation technology: TextToHandSignConverter

[0374] Sign language recognition engine: HandSignRecognition

[0375] Text to Speech Engine: Text to Speech

[0376] Specific examples

[0377] For example, a user might say "Help me, it's urgent" into their smartphone. This speech is captured by a microphone and converted into text "Help me, it's urgent" using Google Speech Recognition. The emotion engine detects the user's high level of urgency and sends it to the server. The server analyzes this information and generates a sign language video that expresses a strong sense of urgency. This sign language video is sent to the device and displayed visually to the user.

[0378] Prompt Sentence Examples

[0379] "The user will input a specific voice to communicate an emergency. Convert this voice into text in real time and then generate a sign language video using an emotion engine. Reflect the user's emotions and express them with enhanced sign language actions."

[0380] This will enable security service users to report situations quickly and accurately in the event of an emergency.

[0381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0382] Step 1:

[0383] Voice input from the user

[0384] The user speaks into the smartphone, saying "Help me, it's urgent." The microphone is used as an input device to capture the voice data. The input data is sent to the device in the form of an audio file.

[0385] Step 2:

[0386] Speech-to-text conversion

[0387] The device sends the captured voice data to the Google Speech Recognition engine to convert the voice to text. The input data is in the form of an audio file, and the output data is the corresponding text "Help, it's urgent." The voice recognition engine analyzes the voice waveform and converts it into a string of characters.

[0388] Step 3:

[0389] Sending text and emotional information

[0390] The device sends the converted text data and emotional information analyzed using Emotion Recognition to the server. The input data is the text data and emotional information extracted from the voice waveform, and the output data is a set of text and emotional information sent to the server.

[0391] Step 4:

[0392] Analyzing text and understanding nuance

[0393] The server uses natural language processing (NLP) technology to analyze the context and nuance of the text based on the received text data and emotional information. The input data is text and emotional information, and the output data is more detailed emotional and nuanced information as the analysis results. The analysis engine extracts the context and emotional strength.

[0394] Step 5:

[0395] Sign language video generation

[0396] Based on the analysis results, the server generates a sign language video using TextToHandSignConverter. The input data is the analyzed text and emotion information, and the output data is a video file containing sign language actions. This sign language video includes actions that emphasize the user's sense of urgency.

[0397] Step 6:

[0398] Sending and displaying sign language videos

[0399] The server sends the generated sign language video to the terminal, and the terminal visually displays the received video to the user. The input data is the sign language video file, and the output data is the image displayed on the terminal's display. The user visually checks the sign language video.

[0400] Step 7:

[0401] Sign language capture

[0402] The user responds by signing "I understand." The camera on the device captures this sign language action. The input data is the camera image, and the output data is a video file containing the sign language action.

[0403] Step 8:

[0404] Sign language to text conversion

[0405] The device analyzes the captured sign language gestures using HandSignRecognition and converts them into the corresponding text "I understand." The input data is a video file containing the sign language gestures, and the output data is text data. The sign language recognition engine analyzes the features of the gestures and converts them into a string of characters.

[0406] Step 9:

[0407] Text to speech conversion

[0408] The terminal converts the converted text into audio data using a TextToSpeech engine and plays it back to the user. The input data is text data, and the output data is audio data. The speech synthesis engine converts the string of characters into audio waveforms.

[0409] This will enable security service users to communicate quickly and accurately even in emergency situations.

[0410] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0411] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0412] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0413] [Second embodiment]

[0414] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0415] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0416] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0417] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0418] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0420] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0421] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0422] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0423] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0424] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0425] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0426] The present invention provides a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. As an embodiment of the present invention, the process of the program is described below in stages.

[0427] Roles of various devices

[0428] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and conversion processing.

[0429] User input

[0430] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the input is done using a keyboard or touch screen.

[0431] Speech recognition and text capture

[0432] For voice input, the device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert speech into text data. For text input, the device uses the text data entered by the user as is.

[0433] Text analysis and nuance understanding

[0434] The device sends the text data to a server for processing. The server uses natural language processing (NLP) techniques to analyze the text data and understand the context, sentiment, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[0435] Sign language video generation

[0436] The server generates appropriate sign language animations based on the analyzed data. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the strength of gratitude.

[0437] Sign language video distribution to users

[0438] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0439] Sign language to text and speech translation

[0440] When a user communicates information using sign language, the device's camera captures the sign, which the device then analyzes using a sign language recognition engine and converts into corresponding text, which it then converts into speech using a speech synthesis engine and plays back to the user.

[0441] Specific example explanation

[0442] For example, if User A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures this sign language action and the server converts it into the corresponding text "You're welcome." The device converts this text into speech and plays it back to User A.

[0443] In this way, the system according to the present invention realizes highly accurate conversion of voice and text input into sign language video, and from sign language to text and voice, supporting smooth communication between users.

[0444] The processing flow will be explained below.

[0445] Step 1:

[0446] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[0447] Step 2:

[0448] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[0449] Step 3:

[0450] The device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert the speech into the text "Thank you." If text is entered, the entered text "Thank you" is used as is.

[0451] Step 4:

[0452] The terminal transmits the acquired text "Thank you" to the server.

[0453] Step 5:

[0454] The server receives the text "Thank you" sent from the terminal.

[0455] Step 6:

[0456] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[0457] Step 7:

[0458] Based on the analysis results, the server generates appropriate sign language video data, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[0459] Step 8:

[0460] The server transmits the generated sign language video data to the terminal.

[0461] Step 9:

[0462] The terminal acquires the sign language video data received from the server.

[0463] Step 10:

[0464] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone.

[0465] Step 11:

[0466] The user signs "hello" to the camera.

[0467] Step 12:

[0468] The device uses a camera to capture the user's sign language actions.

[0469] Step 13:

[0470] The device analyzes the captured sign language gestures using a sign language recognition engine and converts them into the corresponding text "Hello."

[0471] Step 14:

[0472] The device uses a speech synthesis engine (e.g., Text-to-Speech API) to convert the converted text into the voice "Hello" and plays it back to the user.

[0473] In this way, the system can smoothly convert voice and text input into sign language video and sign language back into text and voice, enabling real-time communication between users that includes emotions and nuances.

[0474] Example 1

[0475] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0476] Conventional systems that convert speech or text into sign language may not generate sign language videos accurately. Furthermore, the conversion from sign language to text and speech is also inaccurate, often resulting in poor communication between users. Furthermore, there is a lack of technology that can understand the context and emotions of text and appropriately reflect them in sign language videos.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0478] In this invention, the server includes: means for accepting voice or text input; means for converting the input voice into text; means for analyzing the text and understanding emotions and nuances; means for generating a sign language video based on the analysis results; means for transmitting the generated sign language video to a terminal; means for playing the generated sign language video and visually displaying it to a user; means for capturing a user's sign language movements using a camera; means for analyzing the captured sign language movements and converting them into corresponding text; means for converting the converted text into voice; and means for reflecting the strength and nuance of emotions in the sign language video based on the context and emotions of the analyzed text. This enables highly accurate conversion from voice or text to sign language video and from sign language to text and voice, enabling smooth communication between users.

[0479] "Means for accepting voice or text input" is a general term for devices and software that recognize voice spoken or text entered by a user and incorporate it into the system.

[0480] "Means for converting input speech into text" refers to software or a service that uses a speech recognition engine to analyze the speech uttered by the user and convert it into corresponding text data.

[0481] "Means for analyzing text and understanding emotions and nuances" refers to algorithms or software that use natural language processing technology to analyze text data and extract and understand its context, emotions, and nuances.

[0482] The "means for generating sign language video based on the analysis results" refers to a system that includes a database and video generation technology for generating appropriate sign language video based on the analyzed text data and its emotions and nuances.

[0483] The "means for transmitting the generated sign language video to the terminal" refers to a technology including a communication means and protocol for transferring the generated sign language video data to the user's terminal.

[0484] "Means for playing the generated sign language video and visually displaying it to the user" refers to a display and playback control software that plays the sign language video on the terminal and allows the user to visually confirm the video.

[0485] A "means for capturing a user's sign language actions using a camera" is a device that includes a camera and its control software for recording a user's sign language actions in real time.

[0486] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to a sign language recognition algorithm or software that analyzes video footage of captured sign language actions and generates text data corresponding to those actions.

[0487] "Converted text to speech conversion means" refers to a speech synthesis engine and associated software for converting text data into speech data and playing it back to the user.

[0488] "Means for reflecting emotional strength and nuance in sign language video based on the context and emotion of analyzed text" is a technology for adding appropriate emotional expressions and nuance to sign language video using the context and emotion information of analyzed text data.

[0489] The present invention provides a system for converting voice and text input into sign language video in real time, and then converting sign language into text and voice. The configuration and specific implementation of this system are described in detail below.

[0490] System configuration

[0491] This system mainly consists of three main elements: users, terminals, and servers.

[0492] User operations

[0493] The user inputs voice or text into the terminal using an input device such as a smartphone or microphone. For example, the user may say "Good morning."

[0494] Users can also communicate information using sign language, which is captured by the device's camera.

[0495] Device Features

[0496] The device captures voice data from the user and converts the voice into text data using a speech recognition engine such as the Google Speech-to-Text API. In the case of text input, the device uses the text data entered by the user as is.

[0497] The terminal transmits the acquired text data to the server.

[0498] The device has the ability to receive and play sign language videos sent from the server.

[0499] The device captures the user's sign language input with a camera, analyzes the data with a sign language recognition engine, converts it into text, and then converts the converted text into speech using a speech synthesis engine, which plays it back to the user.

[0500] Server Features

[0501] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) technology, which allows it to understand the context, sentiment, and nuances.

[0502] The server generates appropriate sign language animations based on the analysis results, using a database of sign language actions to generate new sign language animations as needed.

[0503] The server transmits the generated sign language video to the terminal.

[0504] Specific example explanation

[0505] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much" using the Google Speech-to-Text API. The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and converts it into the text "You're welcome" using a sign language recognition engine. The device then converts this text into speech using a speech synthesis engine and plays it back to user A. This process enables smooth communication between users.

[0506] Prompt Sentence Examples

[0507] If the user types "hello" via voice or text, explain how you would process this input and convert it into a sign language video, and then from sign language back to text and speech.

[0508] This system converts voice and text input into sign language video, and sign language into text and voice with high accuracy, supporting smooth communication between users.

[0509] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0510] Step 1: User types into terminal

[0511] The user inputs speech or text into the terminal.

[0512] Input: User's voice "Good morning" or text "Good morning".

[0513] Action: A user says "Good morning" to their smartphone or types "Good morning" on their keyboard.

[0514] Output: The captured audio or text data.

[0515] Step 2: Speech recognition or text capture

[0516] The device receives voice data from the user and converts it into text data using the Google Speech-to-Text API.

[0517] Input: Audio data.

[0518] How it works: The device uses a speech recognition engine to convert speech into text, for example, converting "Good morning" into text data.

[0519] Output: Text data "Good morning".

[0520] Step 3: Sending text data to the server

[0521] The terminal transmits the acquired text data to the server.

[0522] Input: Text data "Good morning".

[0523] Operation: The device sends text data to the server over the network.

[0524] Output: Text data passed to the server.

[0525] Step 4: Text Analysis with NLP

[0526] The server analyzes the received text data to understand the context, sentiment, and nuances.

[0527] Input: Received text data "Good morning."

[0528] How it works: The server uses natural language processing techniques to analyze the text. For example, "Good morning" is recognized as a common greeting.

[0529] Output: Analyzed contextual information and sentiment data.

[0530] Step 5: Generate sign language video

[0531] The server generates appropriate sign language video based on the analyzed data.

[0532] Input: Analyzed contextual information and sentiment data.

[0533] Action: The server refers to a database of sign language actions and generates a sign language animation that expresses "good morning."

[0534] Output: The generated sign language video.

[0535] Step 6: Streaming sign language videos to devices

[0536] The server sends the generated sign language video to the terminal.

[0537] Input: Generated sign language video.

[0538] Operation: The server sends video data to the device via the network.

[0539] Output: Sign language video received on the device.

[0540] Step 7: Play and view the sign language video

[0541] The terminal plays the received sign language video and displays it visually to the user.

[0542] Input: Received sign language video data.

[0543] Operation: The device plays the received sign language video and displays it on the display. For example, a sign language video of "Good morning" is displayed to User B.

[0544] Output: Sign language video displayed to the user.

[0545] Step 8: Translate sign language to text and speech

[0546] The user communicates information in sign language.

[0547] Input: The user's sign language gesture.

[0548] How it works: The camera captures sign language gestures, which the device analyzes using a sign language recognition engine and converts into corresponding text, which is then converted into speech using a speech synthesis engine.

[0549] Output: Converted text and audio data. When User A replies "Good morning" in sign language, the text and audio "Good morning" are played back to User A.

[0550] In this way, this system achieves smooth communication between users through each step.

[0551] (Application example 1)

[0552] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Conventional systems for converting speech or text into sign language have had the problem of difficulty in real-time communication between different language modes. Furthermore, converting sign language into text or speech is complex, making smooth communication difficult, especially in face-to-face environments such as brick-and-mortar stores. This has resulted in a lack of support for customers and store staff with disabilities to communicate efficiently.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0555] In this invention, the server includes means for accepting voice or text input, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, and means for playing back the generated sign language video, thereby enabling efficient real-time communication between customers with disabilities and store staff in physical stores.

[0556] "Voice" refers to the ability to electronically record human voices and process them as digital data.

[0557] "Characters" are a collection of symbols that electronically express a person's intentions or information.

[0558] "Means for accepting input" refers to a device or function that receives voice or text from a user and converts that data into a format that can be processed within the system.

[0559] A "speech-to-text means" is a technology that analyzes received speech data and converts it into a corresponding string of written symbols.

[0560] "A means of analyzing characters and understanding emotions and nuances" is a technology that analyzes the usage and structure of words in a sentence and extracts the emotions and subtle meanings behind them.

[0561] "Means for generating sign language images" refers to a technology that uses analyzed text information to create visual data that mimics human hand and body movements and is expressed as sign language.

[0562] "Means for playing the generated sign language video" refers to a technology for visually presenting the generated sign language video to a user using an appropriate display device.

[0563] "Video equipment" refers to cameras and related devices that can capture user movements and process the data in real time.

[0564] "Sign gestures" refer to specific movements and positions of the hands and body for the purpose of communication, thereby creating a visual representation of language.

[0565] "Means for capturing sign language actions" refers to technology that uses a device such as a camera to record a user's sign language actions as digital data.

[0566] The "means for analyzing and converting into corresponding characters" is a technology for analyzing the data of the captured sign language movements and generating a written symbol string corresponding to the movements.

[0567] A "means of converting text into speech" is a technology that analyzes a written string of symbols and expresses that string of symbols in speech.

[0568] "Context" refers to the context of the text and the circumstances surrounding its use, and is an element that forms the meaning of the text as a whole.

[0569] "Emotions" refers to a person's internal sensations and feelings, and focuses on how they affect communication.

[0570] A system for implementing this invention includes a device that accepts voice or text input, a function that converts the input voice into text, a function that analyzes the text and understands emotions and nuances, a function that generates sign language video based on the analysis results, and a function that plays back the generated sign language video.

[0571] The server is responsible for this complex data processing and conversion. Specifically, it uses the Google Speech-to-Text API to acquire audio data. It also uses natural language processing (NLP) technology, such as a deep learning framework like TensorFlow, to analyze text data. It uses the some_sign_language_video_generation_module, which is based on a database of sign language actions, to generate sign language videos.

[0572] The terminal is a device that accepts input from the user and displays the processing results. Voice input is done from a microphone or smartphone, and text input is done from a keyboard or touch screen. Sign language images are displayed using "OpenCV."

[0573] The camera is used to capture sign language movements, which are then converted into text using some_sign_language_recognition_module, which is then converted back into audio using the Google Text-to-Speech API.

[0574] As a concrete example, consider the case where a user asks "Where is this product?" in a physical store. The device captures the voice and converts it into text "Where is this product?" The server analyzes the text to understand the emotion and context and generates a sign language video. The generated sign language video is sent to the device and played back to the user.

[0575] When a store staff member replies in sign language, "This way," the camera captures this sign and converts it into text. The converted text, "This way," is then converted into speech by the server and played back to the user. This process enables efficient real-time communication between customers with disabilities and store staff in physical stores.

[0576] A specific example of a prompt for a generative AI model is, "A customer asks by voice, 'Where is this product?' Please design and implement a system that includes voice capture, text conversion, sign language video generation, sign language recognition, and speech synthesis." This prompt makes the overall system flow easier to understand.

[0577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0578] Step 1:

[0579] The device accepts voice input from the user. When the user says, "Where is this product?", the device's microphone captures the voice. The captured voice data becomes input data. This allows the voice data to be collected and processed in the next step.

[0580] Step 2:

[0581] The device converts the captured voice data into text data. The data is sent to the server, where it is analyzed using the Google Speech-to-Text API, resulting in the text data "Where is this product?" This is the process of speech recognition and speech-to-text conversion.

[0582] Step 3:

[0583] The server analyzes the text data it receives to understand emotions and nuances. It uses natural language processing technology to analyze the context of the sentence and extract the intention and emotion behind the question, "Where is this product?". TensorFlow and other tools are used to analyze the data. As a result, the text and its contextual information are output.

[0584] Step 4:

[0585] The server generates a sign language video based on the analysis results. Using the sign language action database and some_sign_language_video_generation_module, it generates a sign language video corresponding to "Where is this product?" The generated sign language video becomes the output data.

[0586] Step 5:

[0587] The generated sign language video is sent to a terminal for playback. The terminal receives the sign language video and plays it back on a display device. The user can understand the intended question from the displayed sign language video.

[0588] Step 6:

[0589] The user (store staff member) responds in sign language, saying "This way." The camera captures this sign language action in real time. The captured sign language action becomes input data. This records the sign language response as digital data.

[0590] Step 7:

[0591] The server analyzes the sign language movements and converts them into corresponding text data. Using some_sign_language_recognition_module, the captured sign language movements are analyzed and converted into text data such as "This way." This is the step where the sign language movements are extracted as text information.

[0592] Step 8:

[0593] The server converts the text data into audio and plays it back to the user (customer). Using the Google Text-to-Speech API, the text data "This way" is converted into audio data and sent to the device. The device outputs this audio data using a playback device, allowing the user (customer) to receive an audio response.

[0594] This series of processes enables customers with disabilities and store staff to communicate efficiently in real time in physical stores.

[0595] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0596] The present invention is a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. The following describes the program process in stages as an embodiment of the present invention.

[0597] Roles of various devices

[0598] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and data conversion processing.

[0599] User input

[0600] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the user inputs the text using a keyboard or touch screen. Furthermore, by using an emotion engine, the user can include emotions when inputting.

[0601] Speech recognition and text capture

[0602] For voice input, the device uses a voice recognition engine to convert the voice into text data. For text input, the device uses the text data entered by the user as is. The emotion engine recognizes emotions from the user's voice and text, and includes the results in the transmitted data.

[0603] Text analysis and nuance understanding

[0604] The device sends the recognized emotion information along with the text data to a server, which then uses natural language processing (NLP) technology to analyze the text data and understand the context, emotion, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[0605] Sign language video generation

[0606] The server generates appropriate sign language animations based on the analyzed text and emotional information. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the intensity of gratitude.

[0607] Sign language video distribution to users

[0608] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0609] Sign language to text and speech translation

[0610] When a user communicates information using sign language, the device's camera captures the sign. The device analyzes the captured sign using a sign language recognition engine and converts it into corresponding text. The text is then converted into speech using a speech synthesis engine and played back to the user. An emotion engine also operates to extract additional emotional information from the user's facial expressions and actions and reflect it in the speech and text.

[0611] Specific example explanation

[0612] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The emotion engine then detects user A's emotion and determines it to be "strong gratitude." The server analyzes this text and emotion and generates a sign language video expressing strong gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and the server converts it into the corresponding text, "You're welcome." The device converts this text into speech and plays it back as "You're welcome." At the same time, the emotion engine extracts emotion from user B's sign language movements and facial expressions and reflects it in the text and speech.

[0613] In this way, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[0614] The processing flow will be explained below.

[0615] Step 1:

[0616] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[0617] Step 2:

[0618] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[0619] Step 3:

[0620] The device uses a speech recognition engine to convert the speech into the text "Thank you." In the case of text input, the text "Thank you" entered by the user is used as is.

[0621] Step 4:

[0622] The device uses an emotion engine to recognize emotions from the user's voice and text, along with the acquired text "Thank you." The emotion engine extracts emotions from the user's voice intonation and text expression, and transmits the emotion information along with the text data to the server.

[0623] Step 5:

[0624] The server receives the text "Thank you" and the emotion information sent from the terminal.

[0625] Step 6:

[0626] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[0627] Step 7:

[0628] The server generates appropriate sign language video data based on the analysis results, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[0629] Step 8:

[0630] The server transmits the generated sign language video data to the terminal.

[0631] Step 9:

[0632] The terminal acquires the sign language video data received from the server.

[0633] Step 10:

[0634] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone. This sign language video reflects the strength and nuance of the gratitude.

[0635] Step 11:

[0636] The user signs "hello" to the camera.

[0637] Step 12:

[0638] The terminal uses a camera to capture the user's sign language actions.

[0639] Step 13:

[0640] The device analyzes the captured sign language gestures with a sign language recognition engine and converts them into the corresponding text "Hello." At the same time, the emotion engine extracts emotions from the user's facial expressions and sign language gestures and includes the emotion information.

[0641] Step 14:

[0642] The terminal transmits the converted text and emotion information to the server.

[0643] Step 15:

[0644] The server analyzes the received text "Hello" and the emotional information, and generates a corresponding voice. The tone and intensity of the voice are adjusted based on the emotional information.

[0645] Step 16:

[0646] The server transmits the generated voice data to the terminal.

[0647] Step 17:

[0648] The terminal plays back the audio data received from the server. By listening to this audio, the user can understand the meaning and emotion of "hello" expressed in sign language.

[0649] This specific procedure allows the system to smoothly convert voice or text input into sign language video, and sign language into text and voice. Furthermore, an emotion engine is used to enable communication that takes into account the user's emotions, enabling real-time two-way communication that includes emotions and nuances.

[0650] Example 2

[0651] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0652] Conventional communication systems have had issues with not being able to convert speech / text to sign language and vice versa in sufficient real time, resulting in inaccurate transmission of emotions and nuances. Furthermore, when generating sign language videos while taking the user's emotions into account, or converting sign language to text and speech, emotional information is not reflected, making it difficult to achieve natural and rich communication.

[0653] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0654] In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, means for displaying the generated sign language video, means for recognizing the user's emotions and generating the emotions as data, and means for transmitting the emotion data to the server and combining it with the analysis results, thereby enabling real-time communication that includes emotions and nuances.

[0655] A "means for accepting voice or text input" is a device or program that receives voice input and text input from a user and stores them as data.

[0656] A "means for converting input speech to text" is a device or program that uses a speech recognition engine to analyze a speech signal and convert it into a corresponding string of characters.

[0657] "Means for analyzing text and understanding emotions and nuances" refers to a device or program that uses natural language processing technology to analyze the context and emotions of text data and understand its content.

[0658] The "means for generating sign language video based on the analysis results" refers to a device or program that creates appropriate sign language video based on the analyzed text and emotional information.

[0659] The "means for displaying the generated sign language video" refers to a display device or playback application that visually presents the generated sign language video to the user.

[0660] The "means for recognizing a user's emotions and generating those emotions as data" refers to a device or program that uses an emotion recognition engine that analyzes emotions from the user's voice or text and converts them into data.

[0661] The "means for transmitting emotion data to a server and combining it with the analysis results" refers to a device or program that transmits emotion data generated by an emotion recognition engine to a server and integrates it with the text analysis results on the server side.

[0662] The "means for capturing a user's sign language actions using a camera" refers to a device that uses a video camera to capture a user's sign language actions and record them as digital data.

[0663] A "means for analyzing captured sign language actions and converting them into corresponding text" is a device or program that uses a sign language recognition engine to analyze captured sign language actions and generate corresponding text.

[0664] The "converted text to speech conversion means" is a device or program that uses a speech synthesis engine to convert text data into speech data.

[0665] "Means for extracting emotional information from a user's facial expressions and movements and generating it as data" refers to a device or program that uses an emotion recognition engine to analyze a user's facial expressions and movements recognized by a camera or the like and convert those emotions into data.

[0666] The "means for reflecting the generated emotional data in text or voice" refers to a device or program that adjusts the nuances of the text or voice based on the emotional data.

[0667] "Means for referencing the database necessary for generating sign language animation and generating new sign language animation as needed" refers to a device or program that uses a database of sign language actions to generate new or customized sign language animation.

[0668] "Means for transmitting the generated sign language video to a terminal and playing it" refers to a device or program that transmits the sign language video generated by the server to a user's terminal and plays the video on that terminal.

[0669] The present invention is a system that converts voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. Specific embodiments of this system are described in detail below.

[0670] User input

[0671] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "Thank you" by voice, the device captures this voice data using a smartphone or microphone. If the user inputs "Thank you" by text, the device uses a keyboard or touchscreen.

[0672] Speech recognition and text capture

[0673] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text. For text input, the device uses the text entered by the user as is.

[0674] emotion recognition

[0675] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[0676] Data transmission

[0677] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[0678] Text analysis and nuance understanding

[0679] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[0680] Sign language video generation

[0681] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[0682] Sign language video distribution to users

[0683] The sign language video generated by the server is sent to the terminal, which plays the sign language video and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0684] Sign language to text and speech translation

[0685] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[0686] Specific example explanation

[0687] For example, if User A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects User A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from User B's sign language movements and facial expressions and reflects them in the text and speech.

[0688] Prompt Sentence Examples

[0689] An example of a prompt you can use is:

[0690] "Describe a system that converts user-supplied text into sign language video. It includes an emotion engine and understands nuance."

[0691] As a result, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[0692] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0693] Step 1:

[0694] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "thank you" by voice, the device captures this voice data using a smartphone or microphone.

[0695] Input: User speech, text, sign language

[0696] Output: Captured voice data, text data, sign language data

[0697] Step 2:

[0698] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text data.

[0699] Input: Captured audio data

[0700] Output: Text data generated by speech recognition

[0701] Step 3:

[0702] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[0703] Input: Text data, audio data

[0704] Output: Emotion data

[0705] Step 4:

[0706] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[0707] Input: Text data, emotion data

[0708] Output: Data sent to the server

[0709] Step 5:

[0710] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[0711] Input: Received text data, emotion data

[0712] Output: Parsed text data and sentiment data

[0713] Step 6:

[0714] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[0715] Input: Parsed text data and emotion data

[0716] Output: Generated sign language video

[0717] Step 7:

[0718] The server sends the generated sign language video to the device, which then plays it back and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0719] Input: Generated sign language video

[0720] Output: Sign language video sent to the device, user receives visual information

[0721] Step 8:

[0722] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[0723] Input: Captured sign language gestures

[0724] Output: Text data, audio data

[0725] Specifically, for example, if user A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects user A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from user B's sign language movements and facial expressions and reflects them in text and speech.

[0726] (Application example 2)

[0727] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0728] In conventional security services, it has been difficult for hearing-impaired people to quickly and accurately report emergencies and receive appropriate assistance. Furthermore, there is a lack of a means to accurately convey emotions and nuances of urgency when communicating using sign language. This can lead to delays in emergency communications and misunderstandings.

[0729] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating sign language video based on the analysis results, means for displaying the generated sign language video, means for analyzing the user's emotions in an emergency and generating appropriate sign language video, and means for analyzing the user's sign language movements in real time and converting them into voice or text. This enables hearing-impaired people to report emergencies quickly and accurately and receive appropriate support. Furthermore, nuances of emotions and urgency can be accurately conveyed in communication using sign language.

[0730] "Means for accepting voice or text input" refers to a device or interface that a user can use to input voice or text.

[0731] "Means for converting input speech to text" refers to an algorithm or engine that uses speech recognition technology to convert speech data into corresponding text data.

[0732] "Means of analyzing text and understanding emotions and nuances" refers to the function of analyzing text data using natural language processing technology and extracting context, emotions, and nuances.

[0733] "Means for generating sign language videos based on the analysis results" refers to the function for generating appropriate sign language actions based on the analyzed text and emotional information and outputting them in video format.

[0734] "Means for displaying the generated sign language video" refers to hardware and software such as a monitor or display for visually displaying the generated sign language video to the user.

[0735] "Means for analyzing a user's emotions in an emergency and generating appropriate sign language video" refers to a function that quickly analyzes a user's emotions in an emergency and generates emphasized sign language video based on that emotional information.

[0736] "Means for analyzing a user's sign language actions in real time and converting them into speech or text" refers to technologies or algorithms that analyze and convert sign language actions captured using a camera into corresponding speech or text in real time.

[0737] "Means for capturing a user's sign movements using a camera" refers to a camera or related device used to capture the sign movements.

[0738] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to an algorithm or engine for analyzing captured sign language actions and generating text corresponding to those actions.

[0739] "Means for converting converted text into speech" refers to a function for converting text data into speech data using speech synthesis technology.

[0740] "Means for reflecting a user's emotional information in an emergency situation in voice and text" refers to a technology that appropriately reflects emotional information obtained from a user in an emergency situation in the generated voice and text.

[0741] "Means for reflecting emotional strength and nuance in sign language videos based on the context and emotion of the analyzed text" refers to a function that adds emotional strength and nuance to the actions of sign language videos based on the context and emotion information of the analyzed text.

[0742] "Means for adding special nuances to sign language videos through emotion analysis in emergency situations" refers to technology that analyzes particularly heightened emotions in emergency situations and reflects the results in sign language videos to add special nuances.

[0743] This invention is a speech-to-sign language conversion system specialized for security services, which helps hearing-impaired people to communicate quickly and accurately even in emergency situations. The configuration and operation of this system are described in detail below.

[0744] Overall system configuration

[0745] This system consists of a terminal that accepts user input, a server that analyzes the data and generates sign language video, and a device that provides the generated sign language video or audio to the user.

[0746] Device Features

[0747] The user reports an emergency by voice or text. The device is equipped with a voice input device (e.g., microphone) and a text input interface (e.g., keyboard, touchscreen). If the user inputs by voice, the device uses a speech recognition engine (e.g., Google Speech Recognition) to convert the voice into text. Along with the converted text, an emotion engine analyzes the user's emotions and sends that information to the server.

[0748] Server Features

[0749] The server uses natural language processing (NLP) technology to analyze the context and nuances of the text based on the received text and emotional information. For example, for a highly urgent text such as "Help me," it generates a sign language video with a strong emotional expression. A dedicated database and sign language motion generation technology are used to generate the sign language video.

[0750] Displaying the generated sign language video

[0751] The sign language video generated by the server is sent to the device and displayed visually to the user, allowing hearing-impaired people to quickly report their situation and receive appropriate assistance.

[0752] Sign language to text and speech translation

[0753] When a user responds in sign language, the camera on the device captures the sign. The server analyzes the captured sign and converts it into corresponding text. This text is then converted into speech and played back to the user. The sign language recognition engine and speech synthesis engine used are similarly sophisticated.

[0754] The specific hardware and software used

[0755] Audio input device: Microphone

[0756] Text input interface: keyboard, touchscreen

[0757] Speech recognition engine: Google Speech Recognition

[0758] Natural language processing technology: NLP engine

[0759] Emotion Engine: Emotion Recognition

[0760] Sign language motion generation technology: TextToHandSignConverter

[0761] Sign language recognition engine: HandSignRecognition

[0762] Text to Speech Engine: Text to Speech

[0763] Specific examples

[0764] For example, a user might say "Help me, it's urgent" into their smartphone. This speech is captured by a microphone and converted into text "Help me, it's urgent" using Google Speech Recognition. The emotion engine detects the user's high level of urgency and sends it to the server. The server analyzes this information and generates a sign language video that expresses a strong sense of urgency. This sign language video is sent to the device and displayed visually to the user.

[0765] Prompt Sentence Examples

[0766] "The user will input a specific voice to communicate an emergency. Convert this voice into text in real time and then generate a sign language video using an emotion engine. Reflect the user's emotions and express them with enhanced sign language actions."

[0767] This will enable security service users to report situations quickly and accurately in the event of an emergency.

[0768] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0769] Step 1:

[0770] Voice input from the user

[0771] The user speaks into the smartphone, saying "Help me, it's urgent." The microphone is used as an input device to capture the voice data. The input data is sent to the device in the form of an audio file.

[0772] Step 2:

[0773] Speech-to-text conversion

[0774] The device sends the captured voice data to the Google Speech Recognition engine to convert the voice to text. The input data is in the form of an audio file, and the output data is the corresponding text "Help, it's urgent." The voice recognition engine analyzes the voice waveform and converts it into a string of characters.

[0775] Step 3:

[0776] Sending text and emotional information

[0777] The device sends the converted text data and emotional information analyzed using Emotion Recognition to the server. The input data is the text data and emotional information extracted from the voice waveform, and the output data is a set of text and emotional information sent to the server.

[0778] Step 4:

[0779] Analyzing text and understanding nuance

[0780] The server uses natural language processing (NLP) technology to analyze the context and nuance of the text based on the received text data and emotional information. The input data is text and emotional information, and the output data is more detailed emotional and nuanced information as the analysis results. The analysis engine extracts the context and emotional strength.

[0781] Step 5:

[0782] Sign language video generation

[0783] Based on the analysis results, the server generates a sign language video using TextToHandSignConverter. The input data is the analyzed text and emotion information, and the output data is a video file containing sign language actions. This sign language video includes actions that emphasize the user's sense of urgency.

[0784] Step 6:

[0785] Sending and displaying sign language videos

[0786] The server sends the generated sign language video to the terminal, and the terminal visually displays the received video to the user. The input data is the sign language video file, and the output data is the image displayed on the terminal's display. The user visually checks the sign language video.

[0787] Step 7:

[0788] Sign language capture

[0789] The user responds by signing "I understand." The camera on the device captures this sign language action. The input data is the camera image, and the output data is a video file containing the sign language action.

[0790] Step 8:

[0791] Sign language to text conversion

[0792] The device analyzes the captured sign language gestures using HandSignRecognition and converts them into the corresponding text "I understand." The input data is a video file containing the sign language gestures, and the output data is text data. The sign language recognition engine analyzes the features of the gestures and converts them into a string of characters.

[0793] Step 9:

[0794] Text to speech conversion

[0795] The terminal converts the converted text into audio data using a TextToSpeech engine and plays it back to the user. The input data is text data, and the output data is audio data. The speech synthesis engine converts the string of characters into audio waveforms.

[0796] This will enable security service users to communicate quickly and accurately even in emergency situations.

[0797] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0798] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0799] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0800] [Third embodiment]

[0801] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0802] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0803] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0804] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0805] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0806] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0807] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0808] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0809] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0810] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0811] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0812] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0813] The present invention provides a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. As an embodiment of the present invention, the process of the program is described below in stages.

[0814] Roles of various devices

[0815] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and conversion processing.

[0816] User input

[0817] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the input is done using a keyboard or touch screen.

[0818] Speech recognition and text capture

[0819] For voice input, the device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert speech into text data. For text input, the device uses the text data entered by the user as is.

[0820] Text analysis and nuance understanding

[0821] The device sends the text data to a server for processing. The server uses natural language processing (NLP) techniques to analyze the text data and understand the context, sentiment, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[0822] Sign language video generation

[0823] The server generates appropriate sign language animations based on the analyzed data. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the strength of gratitude.

[0824] Sign language video distribution to users

[0825] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0826] Sign language to text and speech translation

[0827] When a user communicates information using sign language, the device's camera captures the sign, which the device then analyzes using a sign language recognition engine and converts into corresponding text, which it then converts into speech using a speech synthesis engine and plays back to the user.

[0828] Specific example explanation

[0829] For example, if User A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures this sign language action and the server converts it into the corresponding text "You're welcome." The device converts this text into speech and plays it back to User A.

[0830] In this way, the system according to the present invention realizes highly accurate conversion of voice and text input into sign language video, and from sign language to text and voice, supporting smooth communication between users.

[0831] The processing flow will be explained below.

[0832] Step 1:

[0833] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[0834] Step 2:

[0835] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[0836] Step 3:

[0837] The device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert the speech into the text "Thank you." If text is entered, the entered text "Thank you" is used as is.

[0838] Step 4:

[0839] The terminal transmits the acquired text "Thank you" to the server.

[0840] Step 5:

[0841] The server receives the text "Thank you" sent from the terminal.

[0842] Step 6:

[0843] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[0844] Step 7:

[0845] Based on the analysis results, the server generates appropriate sign language video data, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[0846] Step 8:

[0847] The server transmits the generated sign language video data to the terminal.

[0848] Step 9:

[0849] The terminal acquires the sign language video data received from the server.

[0850] Step 10:

[0851] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone.

[0852] Step 11:

[0853] The user signs "hello" to the camera.

[0854] Step 12:

[0855] The device uses a camera to capture the user's sign language actions.

[0856] Step 13:

[0857] The device analyzes the captured sign language gestures using a sign language recognition engine and converts them into the corresponding text "Hello."

[0858] Step 14:

[0859] The device uses a speech synthesis engine (e.g., Text-to-Speech API) to convert the converted text into the voice "Hello" and plays it back to the user.

[0860] In this way, the system can smoothly convert voice and text input into sign language video and sign language back into text and voice, enabling real-time communication between users that includes emotions and nuances.

[0861] Example 1

[0862] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0863] Conventional systems that convert speech or text into sign language may not generate sign language videos accurately. Furthermore, the conversion from sign language to text and speech is also inaccurate, often resulting in poor communication between users. Furthermore, there is a lack of technology that can understand the context and emotions of text and appropriately reflect them in sign language videos.

[0864] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0865] In this invention, the server includes: means for accepting voice or text input; means for converting the input voice into text; means for analyzing the text and understanding emotions and nuances; means for generating a sign language video based on the analysis results; means for transmitting the generated sign language video to a terminal; means for playing the generated sign language video and visually displaying it to a user; means for capturing a user's sign language movements using a camera; means for analyzing the captured sign language movements and converting them into corresponding text; means for converting the converted text into voice; and means for reflecting the strength and nuance of emotions in the sign language video based on the context and emotions of the analyzed text. This enables highly accurate conversion from voice or text to sign language video and from sign language to text and voice, enabling smooth communication between users.

[0866] "Means for accepting voice or text input" is a general term for devices and software that recognize voice spoken or text entered by a user and incorporate it into the system.

[0867] "Means for converting input speech into text" refers to software or a service that uses a speech recognition engine to analyze the speech uttered by the user and convert it into corresponding text data.

[0868] "Means for analyzing text and understanding emotions and nuances" refers to algorithms or software that use natural language processing technology to analyze text data and extract and understand its context, emotions, and nuances.

[0869] The "means for generating sign language video based on the analysis results" refers to a system that includes a database and video generation technology for generating appropriate sign language video based on the analyzed text data and its emotions and nuances.

[0870] The "means for transmitting the generated sign language video to the terminal" refers to a technology including a communication means and protocol for transferring the generated sign language video data to the user's terminal.

[0871] "Means for playing the generated sign language video and visually displaying it to the user" refers to a display and playback control software that plays the sign language video on the terminal and allows the user to visually confirm the video.

[0872] A "means for capturing a user's sign language actions using a camera" is a device that includes a camera and its control software for recording a user's sign language actions in real time.

[0873] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to a sign language recognition algorithm or software that analyzes video footage of captured sign language actions and generates text data corresponding to those actions.

[0874] "Converted text to speech conversion means" refers to a speech synthesis engine and associated software for converting text data into speech data and playing it back to the user.

[0875] "Means for reflecting emotional strength and nuance in sign language video based on the context and emotion of analyzed text" is a technology for adding appropriate emotional expressions and nuance to sign language video using the context and emotion information of analyzed text data.

[0876] The present invention provides a system for converting voice and text input into sign language video in real time, and then converting sign language into text and voice. The configuration and specific implementation of this system are described in detail below.

[0877] System configuration

[0878] This system mainly consists of three main elements: users, terminals, and servers.

[0879] User operations

[0880] The user inputs voice or text into the terminal using an input device such as a smartphone or microphone. For example, the user may say "Good morning."

[0881] Users can also communicate information using sign language, which is captured by the device's camera.

[0882] Device Features

[0883] The device captures voice data from the user and converts the voice into text data using a speech recognition engine such as the Google Speech-to-Text API. In the case of text input, the device uses the text data entered by the user as is.

[0884] The terminal transmits the acquired text data to the server.

[0885] The device has the ability to receive and play sign language videos sent from the server.

[0886] The device captures the user's sign language input with a camera, analyzes the data with a sign language recognition engine, converts it into text, and then converts the converted text into speech using a speech synthesis engine, which plays it back to the user.

[0887] Server Features

[0888] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) technology, which allows it to understand the context, sentiment, and nuances.

[0889] The server generates appropriate sign language animations based on the analysis results, using a database of sign language actions to generate new sign language animations as needed.

[0890] The server transmits the generated sign language video to the terminal.

[0891] Specific example explanation

[0892] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much" using the Google Speech-to-Text API. The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and converts it into the text "You're welcome" using a sign language recognition engine. The device then converts this text into speech using a speech synthesis engine and plays it back to user A. This process enables smooth communication between users.

[0893] Prompt Sentence Examples

[0894] If the user types "hello" via voice or text, explain how you would process this input and convert it into a sign language video, and then from sign language back to text and speech.

[0895] This system converts voice and text input into sign language video, and sign language into text and voice with high accuracy, supporting smooth communication between users.

[0896] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0897] Step 1: User types into terminal

[0898] The user inputs speech or text into the terminal.

[0899] Input: User's voice "Good morning" or text "Good morning".

[0900] Action: A user says "Good morning" to their smartphone or types "Good morning" on their keyboard.

[0901] Output: The captured audio or text data.

[0902] Step 2: Speech recognition or text capture

[0903] The device receives voice data from the user and converts it into text data using the Google Speech-to-Text API.

[0904] Input: Audio data.

[0905] How it works: The device uses a speech recognition engine to convert speech into text, for example, converting "Good morning" into text data.

[0906] Output: Text data "Good morning".

[0907] Step 3: Sending text data to the server

[0908] The terminal transmits the acquired text data to the server.

[0909] Input: Text data "Good morning".

[0910] Operation: The device sends text data to the server over the network.

[0911] Output: Text data passed to the server.

[0912] Step 4: Text Analysis with NLP

[0913] The server analyzes the received text data to understand the context, sentiment, and nuances.

[0914] Input: Received text data "Good morning."

[0915] How it works: The server uses natural language processing techniques to analyze the text. For example, "Good morning" is recognized as a common greeting.

[0916] Output: Analyzed contextual information and sentiment data.

[0917] Step 5: Generate sign language video

[0918] The server generates appropriate sign language video based on the analyzed data.

[0919] Input: Analyzed contextual information and sentiment data.

[0920] Action: The server refers to a database of sign language actions and generates a sign language animation that expresses "good morning."

[0921] Output: The generated sign language video.

[0922] Step 6: Streaming sign language videos to devices

[0923] The server sends the generated sign language video to the terminal.

[0924] Input: Generated sign language video.

[0925] Operation: The server sends video data to the device via the network.

[0926] Output: Sign language video received on the device.

[0927] Step 7: Play and view the sign language video

[0928] The terminal plays the received sign language video and displays it visually to the user.

[0929] Input: Received sign language video data.

[0930] Operation: The device plays the received sign language video and displays it on the display. For example, a sign language video of "Good morning" is displayed to User B.

[0931] Output: Sign language video displayed to the user.

[0932] Step 8: Translate sign language to text and speech

[0933] The user communicates information in sign language.

[0934] Input: The user's sign language gesture.

[0935] How it works: The camera captures sign language gestures, which the device analyzes using a sign language recognition engine and converts into corresponding text, which is then converted into speech using a speech synthesis engine.

[0936] Output: Converted text and audio data. When User A replies "Good morning" in sign language, the text and audio "Good morning" are played back to User A.

[0937] In this way, this system achieves smooth communication between users through each step.

[0938] (Application example 1)

[0939] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0940] Conventional systems for converting speech or text into sign language have had the problem of difficulty in real-time communication between different language modes. Furthermore, converting sign language into text or speech is complex, making smooth communication difficult, especially in face-to-face environments such as brick-and-mortar stores. This has resulted in a lack of support for customers and store staff with disabilities to communicate efficiently.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0942] In this invention, the server includes means for accepting voice or text input, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, and means for playing back the generated sign language video, thereby enabling efficient real-time communication between customers with disabilities and store staff in physical stores.

[0943] "Voice" refers to the ability to electronically record human voices and process them as digital data.

[0944] "Characters" are a collection of symbols that electronically express a person's intentions or information.

[0945] "Means for accepting input" refers to a device or function that receives voice or text from a user and converts that data into a format that can be processed within the system.

[0946] A "speech-to-text means" is a technology that analyzes received speech data and converts it into a corresponding string of written symbols.

[0947] "A means of analyzing characters and understanding emotions and nuances" is a technology that analyzes the usage and structure of words in a sentence and extracts the emotions and subtle meanings behind them.

[0948] "Means for generating sign language images" refers to a technology that uses analyzed text information to create visual data that mimics human hand and body movements and is expressed as sign language.

[0949] "Means for playing the generated sign language video" refers to a technology for visually presenting the generated sign language video to a user using an appropriate display device.

[0950] "Video equipment" refers to cameras and related devices that can capture user movements and process the data in real time.

[0951] "Sign gestures" refer to specific movements and positions of the hands and body for the purpose of communication, thereby creating a visual representation of language.

[0952] "Means for capturing sign language actions" refers to technology that uses a device such as a camera to record a user's sign language actions as digital data.

[0953] The "means for analyzing and converting into corresponding characters" is a technology for analyzing the data of the captured sign language movements and generating a written symbol string corresponding to the movements.

[0954] A "means of converting text into speech" is a technology that analyzes a written string of symbols and expresses that string of symbols in speech.

[0955] "Context" refers to the context of the text and the circumstances surrounding its use, and is an element that forms the meaning of the text as a whole.

[0956] "Emotions" refers to a person's internal sensations and feelings, and focuses on how they affect communication.

[0957] A system for implementing this invention includes a device that accepts voice or text input, a function that converts the input voice into text, a function that analyzes the text and understands emotions and nuances, a function that generates sign language video based on the analysis results, and a function that plays back the generated sign language video.

[0958] The server is responsible for this complex data processing and conversion. Specifically, it uses the Google Speech-to-Text API to acquire audio data. It also uses natural language processing (NLP) technology, such as a deep learning framework like TensorFlow, to analyze text data. It uses the some_sign_language_video_generation_module, which is based on a database of sign language actions, to generate sign language videos.

[0959] The terminal is a device that accepts input from the user and displays the processing results. Voice input is done from a microphone or smartphone, and text input is done from a keyboard or touch screen. Sign language images are displayed using "OpenCV."

[0960] The camera is used to capture sign language movements, which are then converted into text using some_sign_language_recognition_module, which is then converted back into audio using the Google Text-to-Speech API.

[0961] As a concrete example, consider the case where a user asks "Where is this product?" in a physical store. The device captures the voice and converts it into text "Where is this product?" The server analyzes the text to understand the emotion and context and generates a sign language video. The generated sign language video is sent to the device and played back to the user.

[0962] When a store staff member replies in sign language, "This way," the camera captures this sign and converts it into text. The converted text, "This way," is then converted into speech by the server and played back to the user. This process enables efficient real-time communication between customers with disabilities and store staff in physical stores.

[0963] A specific example of a prompt for a generative AI model is, "A customer asks by voice, 'Where is this product?' Please design and implement a system that includes voice capture, text conversion, sign language video generation, sign language recognition, and speech synthesis." This prompt makes the overall system flow easier to understand.

[0964] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0965] Step 1:

[0966] The device accepts voice input from the user. When the user says, "Where is this product?", the device's microphone captures the voice. The captured voice data becomes input data. This allows the voice data to be collected and processed in the next step.

[0967] Step 2:

[0968] The device converts the captured voice data into text data. The data is sent to the server, where it is analyzed using the Google Speech-to-Text API, resulting in the text data "Where is this product?" This is the process of speech recognition and speech-to-text conversion.

[0969] Step 3:

[0970] The server analyzes the text data it receives to understand emotions and nuances. It uses natural language processing technology to analyze the context of the sentence and extract the intention and emotion behind the question, "Where is this product?". TensorFlow and other tools are used to analyze the data. As a result, the text and its contextual information are output.

[0971] Step 4:

[0972] The server generates a sign language video based on the analysis results. Using the sign language action database and some_sign_language_video_generation_module, it generates a sign language video corresponding to "Where is this product?" The generated sign language video becomes the output data.

[0973] Step 5:

[0974] The generated sign language video is sent to a terminal for playback. The terminal receives the sign language video and plays it back on a display device. The user can understand the intended question from the displayed sign language video.

[0975] Step 6:

[0976] The user (store staff member) responds in sign language, saying "This way." The camera captures this sign language action in real time. The captured sign language action becomes input data. This records the sign language response as digital data.

[0977] Step 7:

[0978] The server analyzes the sign language movements and converts them into corresponding text data. Using some_sign_language_recognition_module, the captured sign language movements are analyzed and converted into text data such as "This way." This is the step where the sign language movements are extracted as text information.

[0979] Step 8:

[0980] The server converts the text data into audio and plays it back to the user (customer). Using the Google Text-to-Speech API, the text data "This way" is converted into audio data and sent to the device. The device outputs this audio data using a playback device, allowing the user (customer) to receive an audio response.

[0981] This series of processes enables customers with disabilities and store staff to communicate efficiently in real time in physical stores.

[0982] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0983] The present invention is a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. The following describes the program process in stages as an embodiment of the present invention.

[0984] Roles of various devices

[0985] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and data conversion processing.

[0986] User input

[0987] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the user inputs the text using a keyboard or touch screen. Furthermore, by using an emotion engine, the user can include emotions when inputting.

[0988] Speech recognition and text capture

[0989] For voice input, the device uses a voice recognition engine to convert the voice into text data. For text input, the device uses the text data entered by the user as is. The emotion engine recognizes emotions from the user's voice and text, and includes the results in the transmitted data.

[0990] Text analysis and nuance understanding

[0991] The device sends the recognized emotion information along with the text data to a server, which then uses natural language processing (NLP) technology to analyze the text data and understand the context, emotion, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[0992] Sign language video generation

[0993] The server generates appropriate sign language animations based on the analyzed text and emotional information. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the intensity of gratitude.

[0994] Sign language video distribution to users

[0995] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[0996] Sign language to text and speech translation

[0997] When a user communicates information using sign language, the device's camera captures the sign. The device analyzes the captured sign using a sign language recognition engine and converts it into corresponding text. The text is then converted into speech using a speech synthesis engine and played back to the user. An emotion engine also operates to extract additional emotional information from the user's facial expressions and actions and reflect it in the speech and text.

[0998] Specific example explanation

[0999] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The emotion engine then detects user A's emotion and determines it to be "strong gratitude." The server analyzes this text and emotion and generates a sign language video expressing strong gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and the server converts it into the corresponding text, "You're welcome." The device converts this text into speech and plays it back as "You're welcome." At the same time, the emotion engine extracts emotion from user B's sign language movements and facial expressions and reflects it in the text and speech.

[1000] In this way, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[1001] The processing flow will be explained below.

[1002] Step 1:

[1003] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[1004] Step 2:

[1005] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[1006] Step 3:

[1007] The device uses a speech recognition engine to convert the speech into the text "Thank you." In the case of text input, the text "Thank you" entered by the user is used as is.

[1008] Step 4:

[1009] The device uses an emotion engine to recognize emotions from the user's voice and text, along with the acquired text "Thank you." The emotion engine extracts emotions from the user's voice intonation and text expression, and transmits the emotion information along with the text data to the server.

[1010] Step 5:

[1011] The server receives the text "Thank you" and the emotion information sent from the terminal.

[1012] Step 6:

[1013] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[1014] Step 7:

[1015] The server generates appropriate sign language video data based on the analysis results, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[1016] Step 8:

[1017] The server transmits the generated sign language video data to the terminal.

[1018] Step 9:

[1019] The terminal acquires the sign language video data received from the server.

[1020] Step 10:

[1021] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone. This sign language video reflects the strength and nuance of the gratitude.

[1022] Step 11:

[1023] The user signs "hello" to the camera.

[1024] Step 12:

[1025] The terminal uses a camera to capture the user's sign language actions.

[1026] Step 13:

[1027] The device analyzes the captured sign language gestures with a sign language recognition engine and converts them into the corresponding text "Hello." At the same time, the emotion engine extracts emotions from the user's facial expressions and sign language gestures and includes the emotion information.

[1028] Step 14:

[1029] The terminal transmits the converted text and emotion information to the server.

[1030] Step 15:

[1031] The server analyzes the received text "Hello" and the emotional information, and generates a corresponding voice. The tone and intensity of the voice are adjusted based on the emotional information.

[1032] Step 16:

[1033] The server transmits the generated voice data to the terminal.

[1034] Step 17:

[1035] The terminal plays back the audio data received from the server. By listening to this audio, the user can understand the meaning and emotion of "hello" expressed in sign language.

[1036] This specific procedure allows the system to smoothly convert voice or text input into sign language video, and sign language into text and voice. Furthermore, an emotion engine is used to enable communication that takes into account the user's emotions, enabling real-time two-way communication that includes emotions and nuances.

[1037] Example 2

[1038] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1039] Conventional communication systems have had issues with not being able to convert speech / text to sign language and vice versa in sufficient real time, resulting in inaccurate transmission of emotions and nuances. Furthermore, when generating sign language videos while taking the user's emotions into account, or converting sign language to text and speech, emotional information is not reflected, making it difficult to achieve natural and rich communication.

[1040] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1041] In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, means for displaying the generated sign language video, means for recognizing the user's emotions and generating the emotions as data, and means for transmitting the emotion data to the server and combining it with the analysis results, thereby enabling real-time communication that includes emotions and nuances.

[1042] A "means for accepting voice or text input" is a device or program that receives voice input and text input from a user and stores them as data.

[1043] A "means for converting input speech to text" is a device or program that uses a speech recognition engine to analyze a speech signal and convert it into a corresponding string of characters.

[1044] "Means for analyzing text and understanding emotions and nuances" refers to a device or program that uses natural language processing technology to analyze the context and emotions of text data and understand its content.

[1045] The "means for generating sign language video based on the analysis results" refers to a device or program that creates appropriate sign language video based on the analyzed text and emotional information.

[1046] The "means for displaying the generated sign language video" refers to a display device or playback application that visually presents the generated sign language video to the user.

[1047] The "means for recognizing a user's emotions and generating those emotions as data" refers to a device or program that uses an emotion recognition engine that analyzes emotions from the user's voice or text and converts them into data.

[1048] The "means for transmitting emotion data to a server and combining it with the analysis results" refers to a device or program that transmits emotion data generated by an emotion recognition engine to a server and integrates it with the text analysis results on the server side.

[1049] The "means for capturing a user's sign language actions using a camera" refers to a device that uses a video camera to capture a user's sign language actions and record them as digital data.

[1050] A "means for analyzing captured sign language actions and converting them into corresponding text" is a device or program that uses a sign language recognition engine to analyze captured sign language actions and generate corresponding text.

[1051] The "converted text to speech conversion means" is a device or program that uses a speech synthesis engine to convert text data into speech data.

[1052] "Means for extracting emotional information from a user's facial expressions and movements and generating it as data" refers to a device or program that uses an emotion recognition engine to analyze a user's facial expressions and movements recognized by a camera or the like and convert those emotions into data.

[1053] The "means for reflecting the generated emotional data in text or voice" refers to a device or program that adjusts the nuances of the text or voice based on the emotional data.

[1054] "Means for referencing the database necessary for generating sign language animation and generating new sign language animation as needed" refers to a device or program that uses a database of sign language actions to generate new or customized sign language animation.

[1055] "Means for transmitting the generated sign language video to a terminal and playing it" refers to a device or program that transmits the sign language video generated by the server to a user's terminal and plays the video on that terminal.

[1056] The present invention is a system that converts voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. Specific embodiments of this system are described in detail below.

[1057] User input

[1058] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "Thank you" by voice, the device captures this voice data using a smartphone or microphone. If the user inputs "Thank you" by text, the device uses a keyboard or touchscreen.

[1059] Speech recognition and text capture

[1060] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text. For text input, the device uses the text entered by the user as is.

[1061] emotion recognition

[1062] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[1063] Data transmission

[1064] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[1065] Text analysis and nuance understanding

[1066] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[1067] Sign language video generation

[1068] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[1069] Sign language video distribution to users

[1070] The sign language video generated by the server is sent to the terminal, which plays the sign language video and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[1071] Sign language to text and speech translation

[1072] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[1073] Specific example explanation

[1074] For example, if User A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects User A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from User B's sign language movements and facial expressions and reflects them in the text and speech.

[1075] Prompt Sentence Examples

[1076] An example of a prompt you can use is:

[1077] "Describe a system that converts user-supplied text into sign language video. It includes an emotion engine and understands nuance."

[1078] As a result, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[1079] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1080] Step 1:

[1081] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "thank you" by voice, the device captures this voice data using a smartphone or microphone.

[1082] Input: User speech, text, sign language

[1083] Output: Captured voice data, text data, sign language data

[1084] Step 2:

[1085] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text data.

[1086] Input: Captured audio data

[1087] Output: Text data generated by speech recognition

[1088] Step 3:

[1089] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[1090] Input: Text data, audio data

[1091] Output: Emotion data

[1092] Step 4:

[1093] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[1094] Input: Text data, emotion data

[1095] Output: Data sent to the server

[1096] Step 5:

[1097] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[1098] Input: Received text data, emotion data

[1099] Output: Parsed text data and sentiment data

[1100] Step 6:

[1101] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[1102] Input: Parsed text data and emotion data

[1103] Output: Generated sign language video

[1104] Step 7:

[1105] The server sends the generated sign language video to the device, which then plays it back and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[1106] Input: Generated sign language video

[1107] Output: Sign language video sent to the device, user receives visual information

[1108] Step 8:

[1109] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[1110] Input: Captured sign language gestures

[1111] Output: Text data, audio data

[1112] Specifically, for example, if user A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects user A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from user B's sign language movements and facial expressions and reflects them in text and speech.

[1113] (Application example 2)

[1114] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1115] In conventional security services, it has been difficult for hearing-impaired people to quickly and accurately report emergencies and receive appropriate assistance. Furthermore, there is a lack of a means to accurately convey emotions and nuances of urgency when communicating using sign language. This can lead to delays in emergency communications and misunderstandings.

[1116] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating sign language video based on the analysis results, means for displaying the generated sign language video, means for analyzing the user's emotions in an emergency and generating appropriate sign language video, and means for analyzing the user's sign language movements in real time and converting them into voice or text. This enables hearing-impaired people to report emergencies quickly and accurately and receive appropriate support. Furthermore, nuances of emotions and urgency can be accurately conveyed in communication using sign language.

[1117] "Means for accepting voice or text input" refers to a device or interface that a user can use to input voice or text.

[1118] "Means for converting input speech to text" refers to an algorithm or engine that uses speech recognition technology to convert speech data into corresponding text data.

[1119] "Means of analyzing text and understanding emotions and nuances" refers to the function of analyzing text data using natural language processing technology and extracting context, emotions, and nuances.

[1120] "Means for generating sign language videos based on the analysis results" refers to the function for generating appropriate sign language actions based on the analyzed text and emotional information and outputting them in video format.

[1121] "Means for displaying the generated sign language video" refers to hardware and software such as a monitor or display for visually displaying the generated sign language video to the user.

[1122] "Means for analyzing a user's emotions in an emergency and generating appropriate sign language video" refers to a function that quickly analyzes a user's emotions in an emergency and generates emphasized sign language video based on that emotional information.

[1123] "Means for analyzing a user's sign language actions in real time and converting them into speech or text" refers to technologies or algorithms that analyze and convert sign language actions captured using a camera into corresponding speech or text in real time.

[1124] "Means for capturing a user's sign movements using a camera" refers to a camera or related device used to capture the sign movements.

[1125] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to an algorithm or engine for analyzing captured sign language actions and generating text corresponding to those actions.

[1126] "Means for converting converted text into speech" refers to a function for converting text data into speech data using speech synthesis technology.

[1127] "Means for reflecting a user's emotional information in an emergency situation in voice and text" refers to a technology that appropriately reflects emotional information obtained from a user in an emergency situation in the generated voice and text.

[1128] "Means for reflecting emotional strength and nuance in sign language videos based on the context and emotion of the analyzed text" refers to a function that adds emotional strength and nuance to the actions of sign language videos based on the context and emotion information of the analyzed text.

[1129] "Means for adding special nuances to sign language videos through emotion analysis in emergency situations" refers to technology that analyzes particularly heightened emotions in emergency situations and reflects the results in sign language videos to add special nuances.

[1130] This invention is a speech-to-sign language conversion system specialized for security services, which helps hearing-impaired people to communicate quickly and accurately even in emergency situations. The configuration and operation of this system are described in detail below.

[1131] Overall system configuration

[1132] This system consists of a terminal that accepts user input, a server that analyzes the data and generates sign language video, and a device that provides the generated sign language video or audio to the user.

[1133] Device Features

[1134] The user reports an emergency by voice or text. The device is equipped with a voice input device (e.g., microphone) and a text input interface (e.g., keyboard, touchscreen). If the user inputs by voice, the device uses a speech recognition engine (e.g., Google Speech Recognition) to convert the voice into text. Along with the converted text, an emotion engine analyzes the user's emotions and sends that information to the server.

[1135] Server Features

[1136] The server uses natural language processing (NLP) technology to analyze the context and nuances of the text based on the received text and emotional information. For example, for a highly urgent text such as "Help me," it generates a sign language video with a strong emotional expression. A dedicated database and sign language motion generation technology are used to generate the sign language video.

[1137] Displaying the generated sign language video

[1138] The sign language video generated by the server is sent to the device and displayed visually to the user, allowing hearing-impaired people to quickly report their situation and receive appropriate assistance.

[1139] Sign language to text and speech translation

[1140] When a user responds in sign language, the camera on the device captures the sign. The server analyzes the captured sign and converts it into corresponding text. This text is then converted into speech and played back to the user. The sign language recognition engine and speech synthesis engine used are similarly sophisticated.

[1141] The specific hardware and software used

[1142] Audio input device: Microphone

[1143] Text input interface: keyboard, touchscreen

[1144] Speech recognition engine: Google Speech Recognition

[1145] Natural language processing technology: NLP engine

[1146] Emotion Engine: Emotion Recognition

[1147] Sign language motion generation technology: TextToHandSignConverter

[1148] Sign language recognition engine: HandSignRecognition

[1149] Text to Speech Engine: Text to Speech

[1150] Specific examples

[1151] For example, a user might say "Help me, it's urgent" into their smartphone. This speech is captured by a microphone and converted into text "Help me, it's urgent" using Google Speech Recognition. The emotion engine detects the user's high level of urgency and sends it to the server. The server analyzes this information and generates a sign language video that expresses a strong sense of urgency. This sign language video is sent to the device and displayed visually to the user.

[1152] Prompt Sentence Examples

[1153] "The user will input a specific voice to communicate an emergency. Convert this voice into text in real time and then generate a sign language video using an emotion engine. Reflect the user's emotions and express them with enhanced sign language actions."

[1154] This will enable security service users to report situations quickly and accurately in the event of an emergency.

[1155] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1156] Step 1:

[1157] Voice input from the user

[1158] The user speaks into the smartphone, saying "Help me, it's urgent." The microphone is used as an input device to capture the voice data. The input data is sent to the device in the form of an audio file.

[1159] Step 2:

[1160] Speech-to-text conversion

[1161] The device sends the captured voice data to the Google Speech Recognition engine to convert the voice to text. The input data is in the form of an audio file, and the output data is the corresponding text "Help, it's urgent." The voice recognition engine analyzes the voice waveform and converts it into a string of characters.

[1162] Step 3:

[1163] Sending text and emotional information

[1164] The device sends the converted text data and emotional information analyzed using Emotion Recognition to the server. The input data is the text data and emotional information extracted from the voice waveform, and the output data is a set of text and emotional information sent to the server.

[1165] Step 4:

[1166] Analyzing text and understanding nuance

[1167] The server uses natural language processing (NLP) technology to analyze the context and nuance of the text based on the received text data and emotional information. The input data is text and emotional information, and the output data is more detailed emotional and nuanced information as the analysis results. The analysis engine extracts the context and emotional strength.

[1168] Step 5:

[1169] Sign language video generation

[1170] Based on the analysis results, the server generates a sign language video using TextToHandSignConverter. The input data is the analyzed text and emotion information, and the output data is a video file containing sign language actions. This sign language video includes actions that emphasize the user's sense of urgency.

[1171] Step 6:

[1172] Sending and displaying sign language videos

[1173] The server sends the generated sign language video to the terminal, and the terminal visually displays the received video to the user. The input data is the sign language video file, and the output data is the image displayed on the terminal's display. The user visually checks the sign language video.

[1174] Step 7:

[1175] Sign language capture

[1176] The user responds by signing "I understand." The camera on the device captures this sign language action. The input data is the camera image, and the output data is a video file containing the sign language action.

[1177] Step 8:

[1178] Sign language to text conversion

[1179] The device analyzes the captured sign language gestures using HandSignRecognition and converts them into the corresponding text "I understand." The input data is a video file containing the sign language gestures, and the output data is text data. The sign language recognition engine analyzes the features of the gestures and converts them into a string of characters.

[1180] Step 9:

[1181] Text to speech conversion

[1182] The terminal converts the converted text into audio data using a TextToSpeech engine and plays it back to the user. The input data is text data, and the output data is audio data. The speech synthesis engine converts the string of characters into audio waveforms.

[1183] This will enable security service users to communicate quickly and accurately even in emergency situations.

[1184] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1185] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1186] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1187] [Fourth embodiment]

[1188] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1189] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1190] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1191] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1192] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1193] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1194] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1195] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1196] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1197] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1198] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1199] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1200] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1201] The present invention provides a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. As an embodiment of the present invention, the process of the program is described below in stages.

[1202] Roles of various devices

[1203] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and conversion processing.

[1204] User input

[1205] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the input is done using a keyboard or touch screen.

[1206] Speech recognition and text capture

[1207] For voice input, the device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert speech into text data. For text input, the device uses the text data entered by the user as is.

[1208] Text analysis and nuance understanding

[1209] The device sends the text data to a server for processing. The server uses natural language processing (NLP) techniques to analyze the text data and understand the context, sentiment, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[1210] Sign language video generation

[1211] The server generates appropriate sign language animations based on the analyzed data. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the strength of gratitude.

[1212] Sign language video distribution to users

[1213] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[1214] Sign language to text and speech translation

[1215] When a user communicates information using sign language, the device's camera captures the sign, which the device then analyzes using a sign language recognition engine and converts into corresponding text, which it then converts into speech using a speech synthesis engine and plays back to the user.

[1216] Specific example explanation

[1217] For example, if User A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures this sign language action and the server converts it into the corresponding text "You're welcome." The device converts this text into speech and plays it back to User A.

[1218] In this way, the system according to the present invention realizes highly accurate conversion of voice and text input into sign language video, and from sign language to text and voice, supporting smooth communication between users.

[1219] The processing flow will be explained below.

[1220] Step 1:

[1221] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[1222] Step 2:

[1223] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[1224] Step 3:

[1225] The device uses a speech recognition engine (e.g., Google Speech-to-Text API) to convert the speech into the text "Thank you." If text is entered, the entered text "Thank you" is used as is.

[1226] Step 4:

[1227] The terminal transmits the acquired text "Thank you" to the server.

[1228] Step 5:

[1229] The server receives the text "Thank you" sent from the terminal.

[1230] Step 6:

[1231] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[1232] Step 7:

[1233] Based on the analysis results, the server generates appropriate sign language video data, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[1234] Step 8:

[1235] The server transmits the generated sign language video data to the terminal.

[1236] Step 9:

[1237] The terminal acquires the sign language video data received from the server.

[1238] Step 10:

[1239] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone.

[1240] Step 11:

[1241] The user signs "hello" to the camera.

[1242] Step 12:

[1243] The device uses a camera to capture the user's sign language actions.

[1244] Step 13:

[1245] The device analyzes the captured sign language gestures using a sign language recognition engine and converts them into the corresponding text "Hello."

[1246] Step 14:

[1247] The device uses a speech synthesis engine (e.g., Text-to-Speech API) to convert the converted text into the voice "Hello" and plays it back to the user.

[1248] In this way, the system can smoothly convert voice and text input into sign language video and sign language back into text and voice, enabling real-time communication between users that includes emotions and nuances.

[1249] Example 1

[1250] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1251] Conventional systems that convert speech or text into sign language may not generate sign language videos accurately. Furthermore, the conversion from sign language to text and speech is also inaccurate, often resulting in poor communication between users. Furthermore, there is a lack of technology that can understand the context and emotions of text and appropriately reflect them in sign language videos.

[1252] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1253] In this invention, the server includes: means for accepting voice or text input; means for converting the input voice into text; means for analyzing the text and understanding emotions and nuances; means for generating a sign language video based on the analysis results; means for transmitting the generated sign language video to a terminal; means for playing the generated sign language video and visually displaying it to a user; means for capturing a user's sign language movements using a camera; means for analyzing the captured sign language movements and converting them into corresponding text; means for converting the converted text into voice; and means for reflecting the strength and nuance of emotions in the sign language video based on the context and emotions of the analyzed text. This enables highly accurate conversion from voice or text to sign language video and from sign language to text and voice, enabling smooth communication between users.

[1254] "Means for accepting voice or text input" is a general term for devices and software that recognize voice spoken or text entered by a user and incorporate it into the system.

[1255] "Means for converting input speech into text" refers to software or a service that uses a speech recognition engine to analyze the speech uttered by the user and convert it into corresponding text data.

[1256] "Means for analyzing text and understanding emotions and nuances" refers to algorithms or software that use natural language processing technology to analyze text data and extract and understand its context, emotions, and nuances.

[1257] The "means for generating sign language video based on the analysis results" refers to a system that includes a database and video generation technology for generating appropriate sign language video based on the analyzed text data and its emotions and nuances.

[1258] The "means for transmitting the generated sign language video to the terminal" refers to a technology including a communication means and protocol for transferring the generated sign language video data to the user's terminal.

[1259] "Means for playing the generated sign language video and visually displaying it to the user" refers to a display and playback control software that plays the sign language video on the terminal and allows the user to visually confirm the video.

[1260] A "means for capturing a user's sign language actions using a camera" is a device that includes a camera and its control software for recording a user's sign language actions in real time.

[1261] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to a sign language recognition algorithm or software that analyzes video footage of captured sign language actions and generates text data corresponding to those actions.

[1262] "Converted text to speech conversion means" refers to a speech synthesis engine and associated software for converting text data into speech data and playing it back to the user.

[1263] "Means for reflecting emotional strength and nuance in sign language video based on the context and emotion of analyzed text" is a technology for adding appropriate emotional expressions and nuance to sign language video using the context and emotion information of analyzed text data.

[1264] The present invention provides a system for converting voice and text input into sign language video in real time, and then converting sign language into text and voice. The configuration and specific implementation of this system are described in detail below.

[1265] System configuration

[1266] This system mainly consists of three main elements: users, terminals, and servers.

[1267] User operations

[1268] The user inputs voice or text into the terminal using an input device such as a smartphone or microphone. For example, the user may say "Good morning."

[1269] Users can also communicate information using sign language, which is captured by the device's camera.

[1270] Device Features

[1271] The device captures voice data from the user and converts the voice into text data using a speech recognition engine such as the Google Speech-to-Text API. In the case of text input, the device uses the text data entered by the user as is.

[1272] The terminal transmits the acquired text data to the server.

[1273] The device has the ability to receive and play sign language videos sent from the server.

[1274] The device captures the user's sign language input with a camera, analyzes the data with a sign language recognition engine, converts it into text, and then converts the converted text into speech using a speech synthesis engine, which plays it back to the user.

[1275] Server Features

[1276] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) technology, which allows it to understand the context, sentiment, and nuances.

[1277] The server generates appropriate sign language animations based on the analysis results, using a database of sign language actions to generate new sign language animations as needed.

[1278] The server transmits the generated sign language video to the terminal.

[1279] Specific example explanation

[1280] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much" using the Google Speech-to-Text API. The server analyzes this text and generates a sign language video that emphasizes the expression of gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and converts it into the text "You're welcome" using a sign language recognition engine. The device then converts this text into speech using a speech synthesis engine and plays it back to user A. This process enables smooth communication between users.

[1281] Prompt Sentence Examples

[1282] If the user types "hello" via voice or text, explain how you would process this input and convert it into a sign language video, and then from sign language back to text and speech.

[1283] This system converts voice and text input into sign language video, and sign language into text and voice with high accuracy, supporting smooth communication between users.

[1284] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1285] Step 1: User types into terminal

[1286] The user inputs speech or text into the terminal.

[1287] Input: User's voice "Good morning" or text "Good morning".

[1288] Action: A user says "Good morning" to their smartphone or types "Good morning" on their keyboard.

[1289] Output: The captured audio or text data.

[1290] Step 2: Speech recognition or text capture

[1291] The device receives voice data from the user and converts it into text data using the Google Speech-to-Text API.

[1292] Input: Audio data.

[1293] How it works: The device uses a speech recognition engine to convert speech into text, for example, converting "Good morning" into text data.

[1294] Output: Text data "Good morning".

[1295] Step 3: Sending text data to the server

[1296] The terminal transmits the acquired text data to the server.

[1297] Input: Text data "Good morning".

[1298] Operation: The device sends text data to the server over the network.

[1299] Output: Text data passed to the server.

[1300] Step 4: Text Analysis with NLP

[1301] The server analyzes the received text data to understand the context, sentiment, and nuances.

[1302] Input: Received text data "Good morning."

[1303] How it works: The server uses natural language processing techniques to analyze the text. For example, "Good morning" is recognized as a common greeting.

[1304] Output: Analyzed contextual information and sentiment data.

[1305] Step 5: Generate sign language video

[1306] The server generates appropriate sign language video based on the analyzed data.

[1307] Input: Analyzed contextual information and sentiment data.

[1308] Action: The server refers to a database of sign language actions and generates a sign language animation that expresses "good morning."

[1309] Output: The generated sign language video.

[1310] Step 6: Streaming sign language videos to devices

[1311] The server sends the generated sign language video to the terminal.

[1312] Input: Generated sign language video.

[1313] Operation: The server sends video data to the device via the network.

[1314] Output: Sign language video received on the device.

[1315] Step 7: Play and view the sign language video

[1316] The terminal plays the received sign language video and displays it visually to the user.

[1317] Input: Received sign language video data.

[1318] Operation: The device plays the received sign language video and displays it on the display. For example, a sign language video of "Good morning" is displayed to User B.

[1319] Output: Sign language video displayed to the user.

[1320] Step 8: Translate sign language to text and speech

[1321] The user communicates information in sign language.

[1322] Input: The user's sign language gesture.

[1323] How it works: The camera captures sign language gestures, which the device analyzes using a sign language recognition engine and converts into corresponding text, which is then converted into speech using a speech synthesis engine.

[1324] Output: Converted text and audio data. When User A replies "Good morning" in sign language, the text and audio "Good morning" are played back to User A.

[1325] In this way, this system achieves smooth communication between users through each step.

[1326] (Application example 1)

[1327] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1328] Conventional systems for converting speech or text into sign language have had the problem of difficulty in real-time communication between different language modes. Furthermore, converting sign language into text or speech is complex, making smooth communication difficult, especially in face-to-face environments such as brick-and-mortar stores. This has resulted in a lack of support for customers and store staff with disabilities to communicate efficiently.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1330] In this invention, the server includes means for accepting voice or text input, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, and means for playing back the generated sign language video, thereby enabling efficient real-time communication between customers with disabilities and store staff in physical stores.

[1331] "Voice" refers to the ability to electronically record human voices and process them as digital data.

[1332] "Characters" are a collection of symbols that electronically express a person's intentions or information.

[1333] "Means for accepting input" refers to a device or function that receives voice or text from a user and converts that data into a format that can be processed within the system.

[1334] A "speech-to-text means" is a technology that analyzes received speech data and converts it into a corresponding string of written symbols.

[1335] "A means of analyzing characters and understanding emotions and nuances" is a technology that analyzes the usage and structure of words in a sentence and extracts the emotions and subtle meanings behind them.

[1336] "Means for generating sign language images" refers to a technology that uses analyzed text information to create visual data that mimics human hand and body movements and is expressed as sign language.

[1337] "Means for playing the generated sign language video" refers to a technology for visually presenting the generated sign language video to a user using an appropriate display device.

[1338] "Video equipment" refers to cameras and related devices that can capture user movements and process the data in real time.

[1339] "Sign gestures" refer to specific movements and positions of the hands and body for the purpose of communication, thereby creating a visual representation of language.

[1340] "Means for capturing sign language actions" refers to technology that uses a device such as a camera to record a user's sign language actions as digital data.

[1341] The "means for analyzing and converting into corresponding characters" is a technology for analyzing the data of the captured sign language movements and generating a written symbol string corresponding to the movements.

[1342] A "means of converting text into speech" is a technology that analyzes a written string of symbols and expresses that string of symbols in speech.

[1343] "Context" refers to the context of the text and the circumstances surrounding its use, and is an element that forms the meaning of the text as a whole.

[1344] "Emotions" refers to a person's internal sensations and feelings, and focuses on how they affect communication.

[1345] A system for implementing this invention includes a device that accepts voice or text input, a function that converts the input voice into text, a function that analyzes the text and understands emotions and nuances, a function that generates sign language video based on the analysis results, and a function that plays back the generated sign language video.

[1346] The server is responsible for this complex data processing and conversion. Specifically, it uses the Google Speech-to-Text API to acquire audio data. It also uses natural language processing (NLP) technology, such as a deep learning framework like TensorFlow, to analyze text data. It uses the some_sign_language_video_generation_module, which is based on a database of sign language actions, to generate sign language videos.

[1347] The terminal is a device that accepts input from the user and displays the processing results. Voice input is done from a microphone or smartphone, and text input is done from a keyboard or touch screen. Sign language images are displayed using "OpenCV."

[1348] The camera is used to capture sign language movements, which are then converted into text using some_sign_language_recognition_module, which is then converted back into audio using the Google Text-to-Speech API.

[1349] As a concrete example, consider the case where a user asks "Where is this product?" in a physical store. The device captures the voice and converts it into text "Where is this product?" The server analyzes the text to understand the emotion and context and generates a sign language video. The generated sign language video is sent to the device and played back to the user.

[1350] When a store staff member replies in sign language, "This way," the camera captures this sign and converts it into text. The converted text, "This way," is then converted into speech by the server and played back to the user. This process enables efficient real-time communication between customers with disabilities and store staff in physical stores.

[1351] A specific example of a prompt for a generative AI model is, "A customer asks by voice, 'Where is this product?' Please design and implement a system that includes voice capture, text conversion, sign language video generation, sign language recognition, and speech synthesis." This prompt makes the overall system flow easier to understand.

[1352] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1353] Step 1:

[1354] The device accepts voice input from the user. When the user says, "Where is this product?", the device's microphone captures the voice. The captured voice data becomes input data. This allows the voice data to be collected and processed in the next step.

[1355] Step 2:

[1356] The device converts the captured voice data into text data. The data is sent to the server, where it is analyzed using the Google Speech-to-Text API, resulting in the text data "Where is this product?" This is the process of speech recognition and speech-to-text conversion.

[1357] Step 3:

[1358] The server analyzes the text data it receives to understand emotions and nuances. It uses natural language processing technology to analyze the context of the sentence and extract the intention and emotion behind the question, "Where is this product?". TensorFlow and other tools are used to analyze the data. As a result, the text and its contextual information are output.

[1359] Step 4:

[1360] The server generates a sign language video based on the analysis results. Using the sign language action database and some_sign_language_video_generation_module, it generates a sign language video corresponding to "Where is this product?" The generated sign language video becomes the output data.

[1361] Step 5:

[1362] The generated sign language video is sent to a terminal for playback. The terminal receives the sign language video and plays it back on a display device. The user can understand the intended question from the displayed sign language video.

[1363] Step 6:

[1364] The user (store staff member) responds in sign language, saying "This way." The camera captures this sign language action in real time. The captured sign language action becomes input data. This records the sign language response as digital data.

[1365] Step 7:

[1366] The server analyzes the sign language movements and converts them into corresponding text data. Using some_sign_language_recognition_module, the captured sign language movements are analyzed and converted into text data such as "This way." This is the step where the sign language movements are extracted as text information.

[1367] Step 8:

[1368] The server converts the text data into audio and plays it back to the user (customer). Using the Google Text-to-Speech API, the text data "This way" is converted into audio data and sent to the device. The device outputs this audio data using a playback device, allowing the user (customer) to receive an audio response.

[1369] This series of processes enables customers with disabilities and store staff to communicate efficiently in real time in physical stores.

[1370] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1371] The present invention is a system for converting voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. The following describes the program process in stages as an embodiment of the present invention.

[1372] Roles of various devices

[1373] This system mainly consists of three main components: the user, the terminal, and the server. The user inputs voice or text, or signs. The terminal accepts the input and provides the user with sign language video and audio. The server is responsible for data analysis, sign language video generation, and data conversion processing.

[1374] User input

[1375] When a user speaks "Thank you," the device first captures this voice data. The voice input function is realized by a voice input device (smartphone or microphone). When a user speaks "Thank you," the user inputs the text using a keyboard or touch screen. Furthermore, by using an emotion engine, the user can include emotions when inputting.

[1376] Speech recognition and text capture

[1377] For voice input, the device uses a voice recognition engine to convert the voice into text data. For text input, the device uses the text data entered by the user as is. The emotion engine recognizes emotions from the user's voice and text, and includes the results in the transmitted data.

[1378] Text analysis and nuance understanding

[1379] The device sends the recognized emotion information along with the text data to a server, which then uses natural language processing (NLP) technology to analyze the text data and understand the context, emotion, and nuances. For example, it can analyze the strength of gratitude and the nuance of humility behind the word "thank you."

[1380] Sign language video generation

[1381] The server generates appropriate sign language animations based on the analyzed text and emotional information. This process uses a database of sign language actions and generates new sign language animations as needed. For example, a sign language animation for "thank you" is expressed as an action that emphasizes the intensity of gratitude.

[1382] Sign language video distribution to users

[1383] The generated sign language video is sent from the server to the device, where it is played back and displayed visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[1384] Sign language to text and speech translation

[1385] When a user communicates information using sign language, the device's camera captures the sign. The device analyzes the captured sign using a sign language recognition engine and converts it into corresponding text. The text is then converted into speech using a speech synthesis engine and played back to the user. An emotion engine also operates to extract additional emotional information from the user's facial expressions and actions and reflect it in the speech and text.

[1386] Specific example explanation

[1387] For example, if user A speaks "Thank you very much," the device captures this speech and converts it into the text "Thank you very much." The emotion engine then detects user A's emotion and determines it to be "strong gratitude." The server analyzes this text and emotion and generates a sign language video expressing strong gratitude. The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures this sign language movement and the server converts it into the corresponding text, "You're welcome." The device converts this text into speech and plays it back as "You're welcome." At the same time, the emotion engine extracts emotion from user B's sign language movements and facial expressions and reflects it in the text and speech.

[1388] In this way, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[1389] The processing flow will be explained below.

[1390] Step 1:

[1391] The user speaks "Thank you" into the microphone of the smartphone, or types "Thank you" on the keyboard.

[1392] Step 2:

[1393] The device captures audio input from the microphone (for voice input). For text input, this step is skipped.

[1394] Step 3:

[1395] The device uses a speech recognition engine to convert the speech into the text "Thank you." In the case of text input, the text "Thank you" entered by the user is used as is.

[1396] Step 4:

[1397] The device uses an emotion engine to recognize emotions from the user's voice and text, along with the acquired text "Thank you." The emotion engine extracts emotions from the user's voice intonation and text expression, and transmits the emotion information along with the text data to the server.

[1398] Step 5:

[1399] The server receives the text "Thank you" and the emotion information sent from the terminal.

[1400] Step 6:

[1401] The server uses natural language processing (NLP) models to analyze the context, sentiment, and nuance of the received text, for example, to determine the strength of gratitude or humility behind the word "thank you."

[1402] Step 7:

[1403] The server generates appropriate sign language video data based on the analysis results, including the size and speed of the movements to reflect the strength and nuance of the gratitude.

[1404] Step 8:

[1405] The server transmits the generated sign language video data to the terminal.

[1406] Step 9:

[1407] The terminal acquires the sign language video data received from the server.

[1408] Step 10:

[1409] The device plays the sign language video and visually displays it to the user. For example, the user sees the sign language video "Thank you" on the screen of their smartphone. This sign language video reflects the strength and nuance of the gratitude.

[1410] Step 11:

[1411] The user signs "hello" to the camera.

[1412] Step 12:

[1413] The terminal uses a camera to capture the user's sign language actions.

[1414] Step 13:

[1415] The device analyzes the captured sign language gestures with a sign language recognition engine and converts them into the corresponding text "Hello." At the same time, the emotion engine extracts emotions from the user's facial expressions and sign language gestures and includes the emotion information.

[1416] Step 14:

[1417] The terminal transmits the converted text and emotion information to the server.

[1418] Step 15:

[1419] The server analyzes the received text "Hello" and the emotional information, and generates a corresponding voice. The tone and intensity of the voice are adjusted based on the emotional information.

[1420] Step 16:

[1421] The server transmits the generated voice data to the terminal.

[1422] Step 17:

[1423] The terminal plays back the audio data received from the server. By listening to this audio, the user can understand the meaning and emotion of "hello" expressed in sign language.

[1424] This specific procedure allows the system to smoothly convert voice or text input into sign language video, and sign language into text and voice. Furthermore, an emotion engine is used to enable communication that takes into account the user's emotions, enabling real-time two-way communication that includes emotions and nuances.

[1425] Example 2

[1426] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] Conventional communication systems have had issues with not being able to convert speech / text to sign language and vice versa in sufficient real time, resulting in inaccurate transmission of emotions and nuances. Furthermore, when generating sign language videos while taking the user's emotions into account, or converting sign language to text and speech, emotional information is not reflected, making it difficult to achieve natural and rich communication.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1429] In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating a sign language video based on the analysis results, means for displaying the generated sign language video, means for recognizing the user's emotions and generating the emotions as data, and means for transmitting the emotion data to the server and combining it with the analysis results, thereby enabling real-time communication that includes emotions and nuances.

[1430] A "means for accepting voice or text input" is a device or program that receives voice input and text input from a user and stores them as data.

[1431] A "means for converting input speech to text" is a device or program that uses a speech recognition engine to analyze a speech signal and convert it into a corresponding string of characters.

[1432] "Means for analyzing text and understanding emotions and nuances" refers to a device or program that uses natural language processing technology to analyze the context and emotions of text data and understand its content.

[1433] The "means for generating sign language video based on the analysis results" refers to a device or program that creates appropriate sign language video based on the analyzed text and emotional information.

[1434] The "means for displaying the generated sign language video" refers to a display device or playback application that visually presents the generated sign language video to the user.

[1435] The "means for recognizing a user's emotions and generating those emotions as data" refers to a device or program that uses an emotion recognition engine that analyzes emotions from the user's voice or text and converts them into data.

[1436] The "means for transmitting emotion data to a server and combining it with the analysis results" refers to a device or program that transmits emotion data generated by an emotion recognition engine to a server and integrates it with the text analysis results on the server side.

[1437] The "means for capturing a user's sign language actions using a camera" refers to a device that uses a video camera to capture a user's sign language actions and record them as digital data.

[1438] A "means for analyzing captured sign language actions and converting them into corresponding text" is a device or program that uses a sign language recognition engine to analyze captured sign language actions and generate corresponding text.

[1439] The "converted text to speech conversion means" is a device or program that uses a speech synthesis engine to convert text data into speech data.

[1440] "Means for extracting emotional information from a user's facial expressions and movements and generating it as data" refers to a device or program that uses an emotion recognition engine to analyze a user's facial expressions and movements recognized by a camera or the like and convert those emotions into data.

[1441] The "means for reflecting the generated emotional data in text or voice" refers to a device or program that adjusts the nuances of the text or voice based on the emotional data.

[1442] "Means for referencing the database necessary for generating sign language animation and generating new sign language animation as needed" refers to a device or program that uses a database of sign language actions to generate new or customized sign language animation.

[1443] "Means for transmitting the generated sign language video to a terminal and playing it" refers to a device or program that transmits the sign language video generated by the server to a user's terminal and plays the video on that terminal.

[1444] The present invention is a system that converts voice and text input into sign language video in real time, and also has the function of converting sign language into text and voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more natural and nuanced communication is realized. Specific embodiments of this system are described in detail below.

[1445] User input

[1446] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "Thank you" by voice, the device captures this voice data using a smartphone or microphone. If the user inputs "Thank you" by text, the device uses a keyboard or touchscreen.

[1447] Speech recognition and text capture

[1448] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text. For text input, the device uses the text entered by the user as is.

[1449] emotion recognition

[1450] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[1451] Data transmission

[1452] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[1453] Text analysis and nuance understanding

[1454] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[1455] Sign language video generation

[1456] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[1457] Sign language video distribution to users

[1458] The sign language video generated by the server is sent to the terminal, which plays the sign language video and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[1459] Sign language to text and speech translation

[1460] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[1461] Specific example explanation

[1462] For example, if User A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects User A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to User B. If User B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from User B's sign language movements and facial expressions and reflects them in the text and speech.

[1463] Prompt Sentence Examples

[1464] An example of a prompt you can use is:

[1465] "Describe a system that converts user-supplied text into sign language video. It includes an emotion engine and understands nuance."

[1466] As a result, the system of the present invention achieves highly accurate conversion of voice or text input into sign language video, and from sign language to text and voice, supporting real-time communication between users that includes emotions and nuances.

[1467] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1468] Step 1:

[1469] The user uses the device to input information using voice, text, or sign language. For example, if the user inputs "thank you" by voice, the device captures this voice data using a smartphone or microphone.

[1470] Input: User speech, text, sign language

[1471] Output: Captured voice data, text data, sign language data

[1472] Step 2:

[1473] The device uses a speech recognition engine, such as the Google Cloud Speech-to-Text API, to convert the captured audio into text data.

[1474] Input: Captured audio data

[1475] Output: Text data generated by speech recognition

[1476] Step 3:

[1477] The device uses an emotion engine to recognize emotions from the user's voice and text. This process involves algorithms to analyze the tone of the voice and the context of the text, and generates the results as emotion data. For example, if a user says "Thank you very much" with strong gratitude, this will be recognized as emotion data.

[1478] Input: Text data, audio data

[1479] Output: Emotion data

[1480] Step 4:

[1481] The device sends the acquired text data and emotion data to a server using a secure communication protocol such as HTTPS.

[1482] Input: Text data, emotion data

[1483] Output: Data sent to the server

[1484] Step 5:

[1485] The server analyzes the text and emotion data received from the device, using natural language processing (NLP) techniques to understand the context, emotional intensity, and nuances of the emotion. For example, a generative AI model such as GPT-3 is used for this purpose.

[1486] Input: Received text data, emotion data

[1487] Output: Parsed text data and sentiment data

[1488] Step 6:

[1489] The server generates appropriate sign language animations based on the analyzed text and emotional information. A database of sign language movements is referenced, and new sign language animations are generated as needed using 3D modeling tools such as Unity or Blender. For example, a sign language animation expressing "thank you" is generated with movements that emphasize the intensity of gratitude.

[1490] Input: Parsed text data and emotion data

[1491] Output: Generated sign language video

[1492] Step 7:

[1493] The server sends the generated sign language video to the device, which then plays it back and displays it visually to the user. By watching the sign language video, the user can understand the corresponding audio and text content.

[1494] Input: Generated sign language video

[1495] Output: Sign language video sent to the device, user receives visual information

[1496] Step 8:

[1497] When a user communicates information using sign language, the device's camera captures the sign. The device uses a sign language recognition engine, such as AWS Rekognition, to analyze the captured sign and convert it into text. The device then uses a speech synthesis engine, such as the Google Text-to-Speech API, to convert the text into speech, which the device plays back to the user.

[1498] Input: Captured sign language gestures

[1499] Output: Text data, audio data

[1500] Specifically, for example, if user A speaks "Thank you very much," the device captures this speech and converts it to "Thank you very much" using the Google Cloud Speech-to-Text API. The emotion engine detects user A's emotion and determines it to be "strong gratitude." The server analyzes this using GPT-3 and generates a sign language video expressing strong gratitude (Unity is used to generate the sign language video). The generated sign language video is sent to the device, which plays it and shows it to user B. If user B responds in sign language with "You're welcome," the device's camera captures the sign language movements and AWS Rekognition converts it to "You're welcome." The device converts this into speech using the Google Text-to-Speech API and plays it back as "You're welcome." At the same time, the emotion engine extracts emotions from user B's sign language movements and facial expressions and reflects them in text and speech.

[1501] (Application example 2)

[1502] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1503] In conventional security services, it has been difficult for hearing-impaired people to quickly and accurately report emergencies and receive appropriate assistance. Furthermore, there is a lack of a means to accurately convey emotions and nuances of urgency when communicating using sign language. This can lead to delays in emergency communications and misunderstandings.

[1504] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting input of voice or text, means for converting the input voice into text, means for analyzing the text and understanding emotions and nuances, means for generating sign language video based on the analysis results, means for displaying the generated sign language video, means for analyzing the user's emotions in an emergency and generating appropriate sign language video, and means for analyzing the user's sign language movements in real time and converting them into voice or text. This enables hearing-impaired people to report emergencies quickly and accurately and receive appropriate support. Furthermore, nuances of emotions and urgency can be accurately conveyed in communication using sign language.

[1505] "Means for accepting voice or text input" refers to a device or interface that a user can use to input voice or text.

[1506] "Means for converting input speech to text" refers to an algorithm or engine that uses speech recognition technology to convert speech data into corresponding text data.

[1507] "Means of analyzing text and understanding emotions and nuances" refers to the function of analyzing text data using natural language processing technology and extracting context, emotions, and nuances.

[1508] "Means for generating sign language videos based on the analysis results" refers to the function for generating appropriate sign language actions based on the analyzed text and emotional information and outputting them in video format.

[1509] "Means for displaying the generated sign language video" refers to hardware and software such as a monitor or display for visually displaying the generated sign language video to the user.

[1510] "Means for analyzing a user's emotions in an emergency and generating appropriate sign language video" refers to a function that quickly analyzes a user's emotions in an emergency and generates emphasized sign language video based on that emotional information.

[1511] "Means for analyzing a user's sign language actions in real time and converting them into speech or text" refers to technologies or algorithms that analyze and convert sign language actions captured using a camera into corresponding speech or text in real time.

[1512] "Means for capturing a user's sign movements using a camera" refers to a camera or related device used to capture the sign movements.

[1513] "Means for analyzing captured sign language actions and converting them into corresponding text" refers to an algorithm or engine for analyzing captured sign language actions and generating text corresponding to those actions.

[1514] "Means for converting converted text into speech" refers to a function for converting text data into speech data using speech synthesis technology.

[1515] "Means for reflecting a user's emotional information in an emergency situation in voice and text" refers to a technology that appropriately reflects emotional information obtained from a user in an emergency situation in the generated voice and text.

[1516] "Means for reflecting emotional strength and nuance in sign language videos based on the context and emotion of the analyzed text" refers to a function that adds emotional strength and nuance to the actions of sign language videos based on the context and emotion information of the analyzed text.

[1517] "Means for adding special nuances to sign language videos through emotion analysis in emergency situations" refers to technology that analyzes particularly heightened emotions in emergency situations and reflects the results in sign language videos to add special nuances.

[1518] This invention is a speech-to-sign language conversion system specialized for security services, which helps hearing-impaired people to communicate quickly and accurately even in emergency situations. The configuration and operation of this system are described in detail below.

[1519] Overall system configuration

[1520] This system consists of a terminal that accepts user input, a server that analyzes the data and generates sign language video, and a device that provides the generated sign language video or audio to the user.

[1521] Device Features

[1522] The user reports an emergency by voice or text. The device is equipped with a voice input device (e.g., microphone) and a text input interface (e.g., keyboard, touchscreen). If the user inputs by voice, the device uses a speech recognition engine (e.g., Google Speech Recognition) to convert the voice into text. Along with the converted text, an emotion engine analyzes the user's emotions and sends that information to the server.

[1523] Server Features

[1524] The server uses natural language processing (NLP) technology to analyze the context and nuances of the text based on the received text and emotional information. For example, for a highly urgent text such as "Help me," it generates a sign language video with a strong emotional expression. A dedicated database and sign language motion generation technology are used to generate the sign language video.

[1525] Displaying the generated sign language video

[1526] The sign language video generated by the server is sent to the device and displayed visually to the user, allowing hearing-impaired people to quickly report their situation and receive appropriate assistance.

[1527] Sign language to text and speech translation

[1528] When a user responds in sign language, the camera on the device captures the sign. The server analyzes the captured sign and converts it into corresponding text. This text is then converted into speech and played back to the user. The sign language recognition engine and speech synthesis engine used are similarly sophisticated.

[1529] The specific hardware and software used

[1530] Audio input device: Microphone

[1531] Text input interface: keyboard, touchscreen

[1532] Speech recognition engine: Google Speech Recognition

[1533] Natural language processing technology: NLP engine

[1534] Emotion Engine: Emotion Recognition

[1535] Sign language motion generation technology: TextToHandSignConverter

[1536] Sign language recognition engine: HandSignRecognition

[1537] Text to Speech Engine: Text to Speech

[1538] Specific examples

[1539] For example, a user might say "Help me, it's urgent" into their smartphone. This speech is captured by a microphone and converted into text "Help me, it's urgent" using Google Speech Recognition. The emotion engine detects the user's high level of urgency and sends it to the server. The server analyzes this information and generates a sign language video that expresses a strong sense of urgency. This sign language video is sent to the device and displayed visually to the user.

[1540] Prompt Sentence Examples

[1541] "The user will input a specific voice to communicate an emergency. Convert this voice into text in real time and then generate a sign language video using an emotion engine. Reflect the user's emotions and express them with enhanced sign language actions."

[1542] This will enable security service users to report situations quickly and accurately in the event of an emergency.

[1543] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1544] Step 1:

[1545] Voice input from the user

[1546] The user speaks into the smartphone, saying "Help me, it's urgent." The microphone is used as an input device to capture the voice data. The input data is sent to the device in the form of an audio file.

[1547] Step 2:

[1548] Speech-to-text conversion

[1549] The device sends the captured voice data to the Google Speech Recognition engine to convert the voice to text. The input data is in the form of an audio file, and the output data is the corresponding text "Help, it's urgent." The voice recognition engine analyzes the voice waveform and converts it into a string of characters.

[1550] Step 3:

[1551] Sending text and emotional information

[1552] The device sends the converted text data and emotional information analyzed using Emotion Recognition to the server. The input data is the text data and emotional information extracted from the voice waveform, and the output data is a set of text and emotional information sent to the server.

[1553] Step 4:

[1554] Analyzing text and understanding nuance

[1555] The server uses natural language processing (NLP) technology to analyze the context and nuance of the text based on the received text data and emotional information. The input data is text and emotional information, and the output data is more detailed emotional and nuanced information as the analysis results. The analysis engine extracts the context and emotional strength.

[1556] Step 5:

[1557] Sign language video generation

[1558] Based on the analysis results, the server generates a sign language video using TextToHandSignConverter. The input data is the analyzed text and emotion information, and the output data is a video file containing sign language actions. This sign language video includes actions that emphasize the user's sense of urgency.

[1559] Step 6:

[1560] Sending and displaying sign language videos

[1561] The server sends the generated sign language video to the terminal, and the terminal visually displays the received video to the user. The input data is the sign language video file, and the output data is the image displayed on the terminal's display. The user visually checks the sign language video.

[1562] Step 7:

[1563] Sign language capture

[1564] The user responds by signing "I understand." The camera on the device captures this sign language action. The input data is the camera image, and the output data is a video file containing the sign language action.

[1565] Step 8:

[1566] Sign language to text conversion

[1567] The device analyzes the captured sign language gestures using HandSignRecognition and converts them into the corresponding text "I understand." The input data is a video file containing the sign language gestures, and the output data is text data. The sign language recognition engine analyzes the features of the gestures and converts them into a string of characters.

[1568] Step 9:

[1569] Text to speech conversion

[1570] The terminal converts the converted text into audio data using a TextToSpeech engine and plays it back to the user. The input data is text data, and the output data is audio data. The speech synthesis engine converts the string of characters into audio waveforms.

[1571] This will enable security service users to communicate quickly and accurately even in emergency situations.

[1572] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1573] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1574] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1575] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1576] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1577] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1578] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1579] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1580] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1581] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1582] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1583] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1584] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1585] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1586] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1587] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1588] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1589] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1590] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1591] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1592] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1593] The following is further disclosed regarding the above embodiment.

[1594] (Claim 1)

[1595] means for accepting voice or text input;

[1596] a means for converting input speech into text;

[1597] A means of analyzing text and understanding emotions and nuances,

[1598] a means for generating a sign language video based on the analysis result;

[1599] a means for displaying the generated sign language video;

[1600] A system including:

[1601] (Claim 2)

[1602] means for capturing a user's sign language actions using a camera;

[1603] means for analyzing and converting the captured sign language actions into corresponding text;

[1604] means for converting the converted text into speech;

[1605] The system of claim 1 further comprising:

[1606] (Claim 3)

[1607] A method to reflect the strength and nuance of emotions in sign language videos based on the context and emotions of the analyzed text,

[1608] The system of claim 1 further comprising:

[1609] "Example 1"

[1610] (Claim 1)

[1611] means for accepting voice or text input;

[1612] a means for converting input speech into text;

[1613] A means of analyzing text and understanding emotions and nuances,

[1614] a means for generating a sign language video based on the analysis result;

[1615] means for transmitting the generated sign language video to a terminal;

[1616] a means for playing the generated sign language video and visually displaying it to a user;

[1617] A system including:

[1618] (Claim 2)

[1619] means for capturing a user's sign language actions using a camera;

[1620] means for analyzing and converting the captured sign language actions into corresponding text;

[1621] means for converting the converted text into speech;

[1622] The system of claim 1 further comprising:

[1623] (Claim 3)

[1624] Based on the context and emotions of the analyzed text, we will develop a method to reflect the strength and nuance of emotions in sign language videos.

[1625] The system of claim 1 further comprising:

[1626] "Application Example 1"

[1627] (Claim 1)

[1628] means for accepting voice or text input;

[1629] A means for converting input speech into text;

[1630] A means of analyzing characters and understanding emotions and nuances,

[1631] means for generating a sign language video based on the analysis results;

[1632] a means for displaying the generated sign language video;

[1633] a means for playing the generated sign language video;

[1634] A system including:

[1635] (Claim 2)

[1636] means for capturing sign language movements of a user using a video device;

[1637] means for analyzing the captured sign language actions and converting them into corresponding characters;

[1638] A means for converting the converted characters into speech;

[1639] a means for regenerating the same to a user;

[1640] The system of claim 1 further comprising:

[1641] (Claim 3)

[1642] Based on the context and emotions of the analyzed characters, we will develop a method to reflect the strength and nuance of emotions in sign language videos.

[1643] The system of claim 1 further comprising:

[1644] "Example 2: Combining Emotion Engines"

[1645] (Claim 1)

[1646] means for accepting voice or text input;

[1647] a means for converting input speech into text;

[1648] A means of analyzing text and understanding emotions and nuances,

[1649] a means for generating a sign language video based on the analysis result;

[1650] a means for displaying the generated sign language video;

[1651] means for recognizing a user's emotion and generating emotion data;

[1652] A means for transmitting the emotion data to a server and combining it with the analysis results;

[1653] A system including:

[1654] (Claim 2)

[1655] means for capturing a user's sign language actions using a camera;

[1656] means for analyzing and converting the captured sign language actions into corresponding text;

[1657] means for converting the converted text into speech;

[1658] A means for extracting emotional information from a user's facial expressions and movements and generating the information as data;

[1659] A means for reflecting the generated emotion data in text or voice;

[1660] The system of claim 1 further comprising:

[1661] (Claim 3)

[1662] A method to reflect the strength and nuance of emotions in sign language videos based on the context and emotions of the analyzed text,

[1663] A means for referencing a database required for generating sign language videos and generating new sign language videos as needed;

[1664] A means for transmitting the generated sign language video to a terminal and playing it back;

[1665] The system of claim 1 further comprising:

[1666] "Application example 2 when combining emotion engines"

[1667] (Claim 1)

[1668] means for accepting voice or text input;

[1669] a means for converting input speech into text;

[1670] A means of analyzing text and understanding emotions and nuances,

[1671] a means for generating a sign language video based on the analysis result;

[1672] a means for displaying the generated sign language video;

[1673] A means for analyzing a user's emotions in an emergency situation and generating an appropriate sign language video;

[1674] means for analyzing a user's sign language actions in real time and converting them into speech or text;

[1675] A system including:

[1676] (Claim 2)

[1677] means for capturing a user's sign language actions using a camera;

[1678] means for analyzing and converting the captured sign language actions into corresponding text;

[1679] means for converting the converted text into speech;

[1680] A means for reflecting user's emotional information in an emergency situation into voice and text;

[1681] The system of claim 1 further comprising:

[1682] (Claim 3)

[1683] A method to reflect the strength and nuance of emotions in sign language videos based on the context and emotions of the analyzed text,

[1684] A method for adding special nuance to sign language videos through emotion analysis in emergency situations,

[1685] The system of claim 1 further comprising: [Explanation of symbols]

[1686] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting voice or text input; a means for converting input speech into text; A means of analyzing text and understanding emotions and nuances, a means for generating a sign language video based on the analysis result; a means for displaying the generated sign language video; A system including:

2. means for capturing a user's sign language actions using a camera; means for analyzing and converting the captured sign language actions into corresponding text; means for converting the converted text into speech; The system of claim 1 further comprising:

3. A method to reflect the strength and nuance of emotions in sign language videos based on the context and emotions of the analyzed text, The system of claim 1 further comprising:

Images