System

Abstract

To provide a system.SOLUTION: A system includes: means for capturing a motion of a sign language of a user; means which generates a prompt sentence to be inputted to a generative AI model based on the captured motion of the sign language; means for inputting the generated prompt sentence to the generative AI model and acquiring a result obtained by translating the motion of the sign language captured from the generative AI model into the other representation; and means for displaying the acquired translation result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order for deaf people to communicate smoothly with hearing people, they need to understand sign language. However, since sign language varies by country and region, it is difficult for them to understand different sign languages. It is also difficult for hearing people to understand sign language, so communication between hearing people and hearing people is limited. [Means for solving the problem]

[0005] The present invention provides a system that captures sign language motion and converts it into another sign language or text. This makes it possible to convert, for example, Japanese Sign Language into an international sign language, facilitating communication with deaf people overseas. In addition, by converting the sign language of deaf people into text and displaying it in a format that can be viewed by hearing people, communication between deaf people and hearing people can also be facilitated. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15]FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The system of the present invention captures sign language motions using cameras, sensors, and other means. The captured motion data is analyzed by AI and converted into another sign language or text. For example, it is possible to capture the motion of "hello" in Japanese sign language and convert it into "hello" in international sign language. It is also possible to convert the captured motion into text and display the text "hello." This allows deaf and hearing people to understand each other's statements even when they are participating in the same conference.

[0029] "Example 2"

[0030] The system of the present invention is operated through a user interface. The user makes a sign language motion toward the camera. The system captures the motion and analyzes it using AI. The analysis results are converted into another sign language or text and displayed on the user interface. For example, if a deaf person makes a motion to say "Hello" in Japanese Sign Language, the system captures the motion, converts it into "Hello" in international sign language, and displays it on the screen. Also, if a hearing person says "Hello," the system converts the utterance into text and displays it on the screen.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The user performs a sign language motion using a camera, sensor, or other means.

[0034] Step 2: The system captures the motion data.

[0035] Step 3: The captured motion data is analyzed by AI.

[0036] Step 4: The analysis results are converted into another sign language or text.

[0037] Step 5: The conversion result is displayed to the user.

[0038] "Example 2"

[0039] Step 1: The user performs a sign language motion through the system's user interface.

[0040] Step 2: The system captures the motion data.

[0041] Step 3: The captured motion data is analyzed by AI.

[0042] Step 4: The analysis results are converted into another sign language or text.

[0043] Step 5: The conversion result will be displayed on the user interface.

[0044] Example 1

[0045] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0046] Conventional sign language interpretation systems have difficulty accurately capturing sign language motions and converting them into other sign languages ​​or text. In addition, when deaf and hearing people participate in the same conference, it is difficult for them to understand each other's statements in real time. This has hindered smooth communication.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes a means for capturing sign language motion, a means for preprocessing the captured sign language motion, a means for inputting the preprocessed sign language motion into a generative AI model for analysis, a means for converting the analysis result into another sign language or text, and a means for displaying the converted sign language or text. This makes it possible to accurately capture sign language motion and convert it into another sign language or text. In addition, even if a deaf person and a hearing person are participating in the same conference, they can understand each other's remarks in real time.

[0049] A "means for capturing sign language motion" is a device or method for capturing a user's hand movements using a camera or sensor and collecting sign language motion data.

[0050] The "means for pre-processing the captured sign language motion" refers to a device or method that performs processing such as noise removal, data normalization, and frame interpolation on the captured motion data to improve the quality of the data.

[0051] "Means for inputting preprocessed sign language motion into a generative AI model for analysis" refers to a device or method for inputting preprocessed motion data into an AI model and analyzing sign language motion.

[0052] "Means for converting the analysis results into another sign language or text" refers to a device or method for converting the analysis results of an AI model into another sign language or text.

[0053] A "means for displaying the converted sign language or text" is a device or method that visually displays the converted sign language or text to a user.

[0054] "Japanese Sign Language" is a type of sign language used in Japan, a means of communication using specific hand movements and gestures.

[0055] An "international sign language" is a type of sign language used internationally, which is a common means of communication between sign language users in different countries and regions.

[0056] A "generative AI model" is an artificial intelligence model trained to analyze sign language motion data and convert it into other sign languages ​​or text.

[0057] This invention is a system that uses cameras and sensors to capture sign language motions, collects sign language motion data, analyzes it using AI, and converts it into another sign language or text. A specific embodiment of this system will be described below.

[0058] First, a user activates a camera or a sensor (such as Microsoft (registered trademark) Kinect or Leap Motion) to capture the motion of the sign language, which captures the user's hand movements in real time.

[0059] The device then transmits the captured sign language motion data to the server, which then pre-processes the received motion data, including noise removal, data normalization, frame interpolation, etc., to improve the quality of the data.

[0060] The preprocessed data is input by the server into a generative AI model (e.g., a model built with TENSORFLOW (registered trademark) or PyTorch). The AI ​​model analyzes the sign language motion and obtains an analysis result. The analysis result indicates which sign language the sign language motion corresponds to.

[0061] The server converts the analysis results into another sign language (for example, from Japanese Sign Language to an international sign language) or into text. The converted data is sent to the user's device and displayed. Specifically, the text is displayed on the screen, and the sign language motion is reproduced as animation.

[0062] As a concrete example, consider the case of capturing the motion of "Hello" in Japanese Sign Language. A camera or sensor captures the user's hand movements and sends the data to a server. The server uses an AI model to analyze this motion data and converts it into the international sign language "Hello." At the same time, it also converts it into the text "Hello" and displays it on the user's device.

[0063] Example of a prompt to be input to a generative AI model:

[0064] "Please analyze the motion data of "Hello" in Japanese Sign Language and convert it into "Hello" in International Sign Language. Also, please generate "Hello" as text."

[0065] With this system, even if deaf and hearing people are participating in the same conference, they can understand what each other is saying in real time, which will facilitate smooth communication.

[0066] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0067] Step 1:

[0068] A user activates a camera or sensor. A user activates a camera or sensor (e.g., Microsoft Kinect or Leap Motion) to capture sign motion, which captures the user's hand movements in real time. The input is the user's hand movements, and the output is the captured motion data.

[0069] Step 2:

[0070] The device captures the motion data of sign language. The device captures the user's hand movements in real time through the activated camera or sensor. Specifically, it collects hand position and movement data for each frame. The input is the hand movement data from the camera or sensor, and the output is the captured motion data.

[0071] Step 3:

[0072] The device transmits the captured data to the server. The device transmits the captured motion data to the server. This data includes hand position information, motion vectors, etc. The input is the captured motion data, and the output is the data transmitted to the server.

[0073] Step 4:

[0074] The server preprocesses the motion data. The server preprocesses the motion data it receives. Preprocessing includes noise removal, data normalization, frame interpolation, etc. This improves the quality of the data. The input is the motion data sent to the server, and the output is the preprocessed data.

[0075] Step 5:

[0076] The server inputs the preprocessed data into a generative AI model. The server inputs the preprocessed data into an AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model is trained to analyze sign language motion. The input is the preprocessed motion data, and the output is the analysis result of the AI ​​model.

[0077] Step 6:

[0078] The server obtains the analysis results of the AI ​​model. The server obtains the analysis results from the AI ​​model. The analysis results indicate which sign language the sign language motion corresponds to. The input is the analysis results of the AI ​​model, and the output is the analyzed sign language data.

[0079] Step 7:

[0080] The server converts the analysis results into another sign language or text. Based on the analysis results, the server converts into another sign language (for example, from Japanese Sign Language to international sign language) or text. Specifically, it generates motion data and text data for the corresponding sign language. The input is the analyzed sign language data, and the output is the converted sign language or text data.

[0081] Step 8:

[0082] The server sends the conversion result to the terminal. The server sends the converted sign language or text data to the terminal. The input is the converted sign language or text data, and the output is the data sent to the terminal.

[0083] Step 9:

[0084] The terminal displays the conversion result. The terminal displays the received sign language or text data to the user. Specifically, it displays the text on the screen or reproduces the sign language motion with animation. The input is the sign language or text data sent to the terminal, and the output is the information displayed to the user.

[0085] (Application example 1)

[0086] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] There is a problem that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult for hearing-impaired people to communicate with hearing-nonhearing people who cannot understand sign language, which often causes inconvenience when serving customers or giving directions. In addition, there is a lack of technology to convert sign language into voice or text, making real-time communication difficult.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0089] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for analyzing the captured sign language motion and converting it into voice, and a means for converting voice into text or sign language and displaying it. This enables hearing-impaired people and hearing-normal people to communicate smoothly in a brick-and-mortar store.

[0090] A "means for capturing sign language motion" is a device or technology that uses a camera or sensor to capture sign language movements.

[0091] "Means for converting captured sign language motion into another sign language or text" refers to technology that uses AI or machine learning models to convert captured sign language motion into another sign language format or text.

[0092] A "means for displaying converted signs or text" is a device or technology that uses a display or screen to visually present the converted signs or text.

[0093] "Means of analyzing captured sign language motion and converting it into audio" refers to technology that uses AI and voice synthesis technology to convert captured sign language movements into audio.

[0094] "Means for converting speech into text or sign language and displaying it" refers to a device or technology that uses speech recognition technology to convert speech into text or sign language and display it visually.

[0095] A system for implementing this invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for analyzing the captured sign language motion and converting it into speech, and means for converting speech into text or sign language and displaying it.

[0096] 1. System Programming

[0097] The program for this system is implemented using Python. Specifically, it uses OpenCV to capture sign language motion from the camera, loads the AI ​​model using Keras, and analyzes the sign language motion and converts it into text and audio.

[0098] 2. Program Processing

[0099] The server captures the sign language motion using a camera (e.g. Logitech C920). The captured video is pre-processed using OpenCV and resized to 64x64 pixels. It is then input to an AI model loaded using Keras to analyze the sign language motion. The analysis results are converted into text or audio and displayed or played through a display or speaker.

[0100] 3. Specific Examples

[0101] For example, consider a customer service scenario in a brick-and-mortar store. When a hearing-impaired person wears smart glasses and says "hello" in sign language, the camera captures their motion, and the AI ​​analyzes it to display the text "hello" on the smart glasses' display. Conversely, when a store clerk says "Welcome," the speech is converted into text and displayed as sign language.

[0102] 4. Examples of prompts

[0103] Here is an example of a prompt to be input to the generative AI model:

[0104] "I want to develop a sign language interpretation application. I want to create a system that captures sign language motion with a camera, analyzes it with AI, and converts it into text. Can you please tell me a program that uses Python, OpenCV, and Keras to capture sign language motion and analyze it with an AI model?"

[0105] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0106] Step 1:

[0107] The server captures the sign language motion using a camera. The input is video data from the camera, and the output is the captured sign language motion video. Specifically, the camera captures video in real time and transmits the video to the server frame by frame.

[0108] Step 2:

[0109] The server preprocesses the captured video using OpenCV. The input is the captured sign language motion video, and the output is the preprocessed video data. Specifically, the video is resized to 64x64 pixels and converted into a format that can be input to the AI ​​model.

[0110] Step 3:

[0111] The server inputs the preprocessed video data into the AI ​​model loaded using Keras. The input is the preprocessed video data, and the output is the sign language motion analysis result. In concrete terms, the AI ​​model analyzes the video data and generates prediction results for converting the sign language motion into text and audio.

[0112] Step 4:

[0113] The server converts the analysis results into text or audio. The input is the prediction result of the AI ​​model, and the output is text data or audio data. Specifically, it generates text based on the prediction result, and generates audio using speech synthesis technology if necessary.

[0114] Step 5:

[0115] The server displays or plays the generated text or audio through a display or speaker. The input is text data or audio data, and the output is visual or auditory information. As a specific operation, the text is displayed on the display and the audio is played from the speaker.

[0116] Step 6:

[0117] The user sees the text shown on the display or the audio played from the speaker. The input is the visual or auditory information, and the output is the user's understanding. In concrete terms, the user reads the displayed text or listens to the played audio.

[0118] Example 2

[0119] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] Conventional sign language recognition systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. They also lacked the functionality to convert speech into text and display it in a format that can be read by hearing-impaired people. This has led to problems with smooth communication between hearing-impaired people and hearing-able people.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0122] In this invention, the server includes means for capturing sign language motion, means for analyzing the captured sign language motion, means for converting the analyzed sign language motion into another sign language or text, means for displaying the converted sign language or text, means for capturing voice, means for converting the captured voice into text, and means for displaying the converted text. This makes it possible to accurately analyze sign language motion and convert it into another sign language or text, and further convert voice into text and display it, enabling smooth communication between hearing-impaired people and hearing-normal people.

[0123] The "means for capturing sign language motion" refers to a device or method for capturing the sign language movements made by a user as video data using a photographing device such as a camera.

[0124] "Means for analyzing the captured sign language motion" refers to an AI model or algorithm for analyzing the captured video data and identifying the content of the sign language.

[0125] "Means for converting analyzed sign language motion into another sign language or text" refers to a device or method for converting the sign language content into another sign language or text format based on the analysis results.

[0126] A "means for displaying the converted sign language or text" is a display or software for displaying the converted sign language or text in a user interface.

[0127] The "means for capturing voice" refers to a device or method for capturing a user's speech as voice data using a voice input device such as a microphone.

[0128] "Means for converting captured speech into text" refers to speech recognition software or algorithms used to analyze the captured speech data and convert it into text form.

[0129] "Means for displaying the converted text" refers to a display or software for displaying the converted text on a user interface.

[0130] This invention is a system that captures sign language motion, analyzes it, and converts it into another sign language or text. It also includes a function to capture voice, convert it into text, and display it. A specific embodiment of this system will be described below.

[0131] Hardware and Software Used

[0132] Hardware:

[0133] Camera: A high-resolution camera is used to capture the user's sign language motions.

[0134] Microphone: An audio input device for capturing the user's voice.

[0135] Display: A screen for displaying the user interface.

[0136] software:

[0137] AI model for sign language recognition: Analyzes sign language motions using machine learning frameworks such as TensorFlow and PyTorch.

[0138] Speech Recognition Software: Convert speech to text using speech recognition technology such as Google® Speech-to-Text.

[0139] System Operation

[0140] Sign Language Motion Capture and Analysis:

[0141] When a user performs a sign language motion in front of the camera, the device captures the motion. The captured video data is sent to a server, which analyzes the video data using an AI model for sign language recognition. The analysis results are converted into another sign language or text and sent to the device. The device displays the conversion results on the user interface.

[0142] Audio capture and conversion:

[0143] When a user speaks, the device captures the speech using a microphone. The captured speech data is sent to a server, which uses voice recognition software to convert the speech into text. The converted text is sent to the device and displayed on a user interface.

[0144] Examples

[0145] Sign language motion capture and conversion:

[0146] The user makes the motion to say "hello" in Japanese sign language in front of the camera.

[0147] The device captures the motion using a camera and transmits the video data to a server.

[0148] The server inputs the video data into an AI model for sign language recognition and analyzes it.

[0149] The server converts the analysis result into the international sign language "hello" and sends it to the terminal.

[0150] The terminal displays the conversion result on the user interface.

[0151] Audio capture and conversion:

[0152] The user says "Hello."

[0153] The terminal captures the audio using a microphone and transmits the audio data to the server.

[0154] The server inputs the voice data into speech recognition software and converts it into text.

[0155] The server transmits the conversion result to the terminal.

[0156] The terminal displays the conversion result on the user interface.

[0157] Examples of prompt statements

[0158] "If I make a motion to say 'hello' in Japanese sign language, please convert it into international sign language."

[0159] "If you say 'hello' by voice, convert that speech to text."

[0160] In this way, the server and the terminal work together to realize a system that analyzes the user's sign language and voice, converts it into an appropriate format, and displays it.

[0161] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0162] Step 1:

[0163] The user makes sign language motions towards the camera.

[0164] As a specific action, a user makes a motion to say "hello" in Japanese sign language in front of the camera. The input is the user's sign language motion, and the output is the video data captured by the camera.

[0165] Step 2:

[0166] The device uses a camera to capture sign language motions.

[0167] Specifically, the device uses a high-resolution camera to capture the user's sign language motion as video data. The input is the user's sign language motion, and the output is the captured video data.

[0168] Step 3:

[0169] The terminal transmits the captured video data to the server.

[0170] In particular, the terminal transmits video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[0171] Step 4:

[0172] The server inputs the video data into an AI model for sign language recognition.

[0173] Specifically, the server inputs the received video data into an AI model for sign language recognition (e.g., TensorFlow or PyTorch). The input is the video data sent to the server, and the output is the data input into the AI ​​model.

[0174] Step 5:

[0175] The server uses an AI model to analyze sign language motions.

[0176] Specifically, the server uses the AI ​​model to analyze the video data and identify the sign language content. The input is the video data input to the AI ​​model, and the output is the analyzed sign language content.

[0177] Step 6:

[0178] The server converts the analysis results into another sign language or text.

[0179] Specifically, the server converts the sign language content into another sign language (e.g., international sign language) or text format based on the analysis results. The input is the analyzed sign language content, and the output is the converted sign language or text.

[0180] Step 7:

[0181] The server transmits the conversion result to the terminal.

[0182] Specifically, the server sends the conversion result to the terminal. The input is the converted sign language or text, and the output is the conversion result sent to the terminal.

[0183] Step 8:

[0184] The terminal displays the conversion result on the user interface.

[0185] As a specific operation, the terminal displays the conversion result on a user interface. The input is the conversion result sent to the terminal, and the output is the sign language or text displayed on the user interface.

[0186] Step 9:

[0187] The user speaks aloud.

[0188] As a specific operation, the user says "Hello." The input is the user's voice, and the output is the voice data captured by the microphone.

[0189] Step 10:

[0190] The device captures audio using a microphone.

[0191] In particular, the terminal uses a microphone to capture the user's voice as voice data, with the input being the user's voice and the output being the captured voice data.

[0192] Step 11:

[0193] The device transmits the captured audio data to the server.

[0194] In particular, the terminal transmits voice data to the server in real time, where the input is the captured voice data and the output is the voice data transmitted to the server.

[0195] Step 12:

[0196] The server inputs the voice data into voice recognition software.

[0197] Specifically, the server inputs the received voice data into speech recognition software (e.g. Google Speech-to-Text). The input is the voice data sent to the server, and the output is the data input into the speech recognition software.

[0198] Step 13:

[0199] The server converts the voice data into text.

[0200] Specifically, the server uses speech recognition software to convert the voice data into text, with the input being the voice data entered into the speech recognition software and the output being the converted text.

[0201] Step 14:

[0202] The server transmits the conversion result to the terminal.

[0203] In concrete operation, the server sends the conversion result to the terminal, where the input is the converted text and the output is the conversion result sent to the terminal.

[0204] Step 15:

[0205] The terminal displays the conversion result on the user interface.

[0206] In particular, the terminal displays the conversion result on a user interface, where the input is the conversion result sent to the terminal and the output is the text displayed on the user interface.

[0207] (Application example 2)

[0208] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional sign language translation systems have limited functionality for converting sign language into text and translating between different sign languages, making it difficult for deaf and hearing people to communicate smoothly. In addition, there is a lack of means for deaf and hearing people to communicate smoothly in physical stores.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0211] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for converting voice into text, a means for displaying the converted text, and a means for translating between different sign languages, thereby enabling deaf and hearing people to communicate smoothly in a brick-and-mortar store.

[0212] A "sign language motion capture means" is a device or method that uses a camera or sensor to capture the sign language movements made by a user.

[0213] A "means for converting captured sign language motion into another sign language or text format" is software or algorithms that analyzes captured sign language motions and converts them into another sign language or text format.

[0214] A "means for displaying the converted sign language or text" is a display or screen that visually displays the converted sign language or text in a user interface.

[0215] A "speech-to-text conversion means" is software or algorithms used to convert speech data into text data using speech recognition technology.

[0216] An "interlanguage translation tool" is a translation algorithm or software that converts from one sign language to another.

[0217] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for converting speech to text, means for displaying the converted text, and means for translating between different sign languages.

[0218] System Programming

[0219] The system program performs the following processing.

[0220] 1. Means of capturing sign language motion:

[0221] The system uses cameras and sensors to capture the sign language movements made by the user, specifically, cameras installed in smartphones and smart glasses.

[0222] 2. How to convert captured sign motion into other signs or text:

[0223] It uses machine learning libraries such as TensorFlow to analyze the captured sign language movements and convert them into other sign languages ​​or text formats, using pre-trained generative AI models.

[0224] 3. Means for displaying converted signs and / or text:

[0225] Use the display of a smartphone or smart glasses to visually display the converted sign language or text on a user interface.

[0226] 4. Ways to convert speech to text:

[0227] The SpeechRecognition library is used to convert voice data into text data using speech recognition technology, specifically, using the microphones on smartphones and smart glasses.

[0228] 5. Means of translation between different sign languages:

[0229] Using translation algorithms or software to convert from one sign language to another, including using generative AI models to translate.

[0230] Processing flow

[0231] The server takes in video from the camera and captures the sign language motion. The captured video is analyzed using TensorFlow and converted into another sign language or text. The converted sign language or text is displayed on the display of the smartphone or smart glasses. Using voice recognition technology, the voice data is converted into text data, which is also displayed. Translation between different sign languages ​​is also performed using generative AI models.

[0232] Examples

[0233] For example, if a deaf customer says "Hello" in sign language, the camera in the smart glasses will capture that motion, and the AI ​​will convert it into text and display it on the store clerk's smartphone. If the store clerk says "Welcome," the speech will be converted into text and displayed on the deaf customer's smart glasses.

[0234] Examples of prompt statements

[0235] "When a user says 'Hello' in sign language, capture that motion with a camera, analyze it with an AI model, and convert it into text as 'Hello.' Also, when a store clerk says 'Welcome,' convert that speech into text and display it."

[0236] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0237] Step 1:

[0238] A user performs a sign language motion. The camera in a smartphone or smart glasses captures the motion. The input is the user's sign language motion, and the output is the captured video data. In concrete terms, the camera continuously shoots video and captures the sign language motion in real time.

[0239] Step 2:

[0240] The server receives the captured video data and performs preprocessing. The input is the captured video data, and the output is the preprocessed image data. The specific operations include resizing the video data, removing noise, and extracting the sign language motion features.

[0241] Step 3:

[0242] The server inputs the preprocessed image data into the generative AI model to analyze the sign language motion. The input is the preprocessed image data, and the output is the analysis result, a text representation of the sign language. In concrete terms, the generative AI model analyzes the characteristics of the sign language motion and generates the corresponding text.

[0243] Step 4:

[0244] The server converts the results of the analysis into another sign language or text. The input is a textual representation of the sign language, and the output is the converted sign language or text. In particular, a translation algorithm converts the textual representation of the sign language into a different sign language or text.

[0245] Step 5:

[0246] The server sends the converted sign language or text to the terminal and displays it. The input is the converted sign language or text, and the output is the information displayed on the terminal's display. The specific operation is to visually display the converted sign language or text on the terminal's display.

[0247] Step 6:

[0248] A user speaks using voice. The microphone in a smartphone or smart glasses captures the voice. The input is the user's voice, and the output is the captured voice data. In concrete terms, the microphone continuously records the voice and captures the voice data in real time.

[0249] Step 7:

[0250] The server receives the captured voice data and performs speech recognition. The input is the captured voice data, and the output is text data that is the result of speech recognition. In concrete terms, the speech recognition algorithm analyzes the voice data and generates corresponding text.

[0251] Step 8:

[0252] The server sends the speech recognition results to the terminal and displays them. The input is the text data of the speech recognition results, and the output is the information displayed on the terminal's display. The specific operation is to visually display the speech recognition results on the terminal's display.

[0253] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0254] "Example 1"

[0255] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes the user's emotion from the captured sign motion. Specifically, the emotion engine determines the likelihood that the user is expressing an emotion such as joy, anger, sadness, surprise, etc. from the speed, strength, and rhythm of the sign motion. The emotion engine adjusts the sign-to-text conversion according to the emotion. For example, if it is determined that the user is expressing an emotion of anger, an expression reflecting that emotion can be included in the text.

[0256] "Example 2"

[0257] In another embodiment of the present invention, an emotion engine is incorporated to recognize the emotion of the user from the captured sign motion and adjust the conversion from Japanese Sign Language to International Sign Language based on the emotion. Specifically, if it is determined that the user is expressing an emotion of joy, an expression reflecting the emotion can be included in the international sign language.

[0258] The process flow of each embodiment will be described below.

[0259] "Example 1"

[0260] Step 1: The system receives sign language motions from the user and uses a camera or other device to

[0261] Capture it using a sensor.

[0262] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[0263] Step 3: The emotion engine adjusts the sign-to-text conversion based on the determined emotion. For example, if it determines that the user is expressing an emotion of anger, it can include a wording that reflects that emotion in the text.

[0264] "Example 2"

[0265] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[0266] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[0267] Step 3: The emotion engine adjusts the conversion from Japanese Sign Language to international sign language based on the determined emotion. For example, if it determines that the user is expressing the emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[0268] Example 1

[0269] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0270] Conventional sign language translation systems were able to capture sign language motions and convert them into other sign languages ​​or text, but they lacked the ability to recognize the user's emotions and adjust the translation results accordingly. This made it difficult to accurately convey the nuances and emotions of sign language, resulting in a decline in the quality of communication.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0272] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion from the captured sign language motion, and a means for adjusting the conversion from sign language to text in accordance with the recognized emotion. This enables sign language translation that reflects the user's emotion, realizing more natural and accurate communication.

[0273] "Means for capturing sign language motion" refers to a device or method that uses a camera or motion sensor to capture a user's sign language movements in real time and obtain them as digital data.

[0274] "Means for converting captured sign language motion into another sign language or text" refers to a device or method that uses an AI analysis engine to analyze acquired sign language motion data and convert it into a corresponding other sign language or text.

[0275] "Means for displaying the converted sign language or text" refers to a device or method, such as a display or monitor, for visually presenting the converted sign language or text to a user.

[0276] The "means for recognizing a user's emotions from the captured sign language motion" refers to an emotion engine or algorithm that analyzes characteristics of the sign language motion, such as speed, strength, and rhythm, and estimates the user's emotions.

[0277] "Means for adjusting the sign language-to-text conversion in response to recognized emotions" refers to an apparatus or method that adjusts the sign language-to-text conversion result based on the user's emotions recognized by an emotion engine, and generates text that reflects the emotions.

[0278] The system of the present invention includes a camera and a motion sensor for capturing the motion of the sign language, an AI analysis engine for analyzing the captured motion data, an emotion engine for recognizing the emotion of the user, and a display for displaying the converted sign language and text. Specific embodiments are described below.

[0279] Hardware and Software Used

[0280] Hardware: High resolution camera, motion sensor, display

[0281] Software: AI analysis engine, emotion engine, sign language conversion algorithm

[0282] Data Import

[0283] When a user signs, the device's camera and motion sensor capture the sign language motion in real time. The camera captures hand movements in high resolution, and the motion sensor records the hand position and movement in detail. The captured motion data is then sent from the device to a server.

[0284] 4. Data Analysis

[0285] The server inputs the received motion data into an AI analysis engine, which uses a deep learning model to analyze the motion data and convert it into the corresponding sign language or text, for example converting "hello" in Japanese sign language to "hello" in international sign language.

[0286] emotion recognition

[0287] The server inputs the analyzed sign language data into the emotion engine. The emotion engine extracts features of the sign language motion, such as speed, strength, and rhythm, to recognize the user's emotion. For example, if the sign language is fast and strong, it determines that the user is expressing the emotion of anger. It adjusts the conversion from sign language to text according to the recognized emotion. For example, if the user is expressing the emotion of joy, it generates the text "Hello!" that reflects that emotion.

[0288] Data Output

[0289] The server sends the converted sign language or text to the terminal, which displays the received data to the user. For example, the text "Hello!" is displayed on the terminal screen.

[0290] Examples

[0291] For example, when a user wants to say "hello" in Japanese Sign Language, the following specific actions are performed:

[0292] 1. The user signs.

[0293] 2. The device's camera and motion sensors capture the user's sign language motions in real time.

[0294] 3. The device transmits the captured motion data to the server.

[0295] 4. The server inputs the received motion data into the AI ​​analysis engine.

[0296] 5. The AI ​​analysis engine analyzes the motion data and converts the Japanese sign language "hello" into the international sign language "hello."

[0297] 6. The server inputs the analyzed sign language data into the emotion engine.

[0298] 7. The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[0299] 8. The emotion engine determines that the user is expressing the emotion of happiness and generates text that reflects that emotion: "Hello!"

[0300] 9. The server sends the converted sign language or text to the terminal.

[0301] 10. The device displays the received data to the user, for example displaying the text "Hello!" on the device screen.

[0302] Examples of prompt statements

[0303] Examples of prompt sentences to be input to a generative AI model include the following:

[0304] "Analyze the sign language motion data captured by the camera and convert the Japanese sign language "Hello" into the international sign language "Hello." Also, recognize the user's emotions, and if the emotion includes joy, display "Hello!" in the text."

[0305] In this way, the system can analyze sign language motions in real time and convert them into appropriate sign language and text.

[0306] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0307] Step 1:

[0308] The user signs.

[0309] As a specific action, the user says "hello" in Japanese Sign Language.

[0310] Input: User's sign language motion

[0311] Output: None

[0312] Step 2:

[0313] The device's camera and motion sensors capture the user's sign language movements in real time.

[0314] Specifically, the camera captures hand movements in high resolution, while the motion sensor records the hand's position and movement in detail.

[0315] Input: User's sign language motion

[0316] Output: Motion data

[0317] Step 3:

[0318] The terminal transmits the captured motion data to the server.

[0319] Specifically, data is transmitted in real time, minimizing latency.

[0320] Input: Motion data

[0321] Output: Motion data sent to the server

[0322] Step 4:

[0323] The server inputs the received motion data into an AI analysis engine.

[0324] Specifically, the AI ​​analytics engine uses deep learning models to analyze motion data.

[0325] Input: Motion data sent to the server

[0326] Output: Analysis results (sign language and text)

[0327] Step 5:

[0328] The server inputs the parsed sign language data into an emotion engine.

[0329] As a specific action, the emotion engine extracts features such as speed, strength and rhythm of sign language motion.

[0330] Input: Analysis results (sign language or text)

[0331] Output: Emotion data

[0332] Step 6:

[0333] The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[0334] Specifically, if the sign language is fast and forceful, it is determined that the user is expressing anger.

[0335] Input: Emotion data

[0336] Output: Recognized emotion

[0337] Step 7:

[0338] The emotion engine adjusts the sign language to text conversion depending on the recognized emotion.

[0339] Specifically, if the user expresses an emotion of joy, the system generates text that reflects that emotion: "Hello!"

[0340] Input: Recognized emotion

[0341] Output: Adjusted text

[0342] Step 8:

[0343] The server sends the converted sign language or text to the terminal.

[0344] Specifically, the data is encrypted and transmitted securely.

[0345] Input: Adjusted text

[0346] Output: Text sent to the terminal

[0347] Step 9:

[0348] The terminal displays the received data to the user.

[0349] The specific action is that the text "Hello!" will be displayed on the device screen.

[0350] Input: Text sent to the terminal

[0351] Output: The displayed text

[0352] (Application example 1)

[0353] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0354] There is an issue that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult to convey information using sign language to hearing-nonhearing people who cannot understand sign language. In addition, there is a lack of technology that can recognize the user's emotions from the motion of sign language and convert it into text or voice that appropriately reflects those emotions, which makes it difficult to communicate accurately with emotions.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0356] In this invention, the server includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotion, and means for reading out the converted text aloud. This enables hearing-impaired people and hearing-normal people to communicate smoothly in physical stores.

[0357] "Sign language motion" refers to the movements of the hands, arms, fingers, etc. when using sign language.

[0358] "Means of capturing" refers to devices or methods that use cameras, sensors, etc. to capture sign language motion as digital data.

[0359] A "conversion method" refers to software or algorithms used to convert captured sign motion into another sign language or text.

[0360] "Display device" means a display or screen for visually presenting the converted sign language or text.

[0361] "Means of recognizing emotions" refers to software or algorithms for analyzing and recognizing a user's emotions from sign language motions.

[0362] "Means for tailoring the conversion" means software or algorithms that appropriately adjust the conversion of sign language to text based on recognized emotions.

[0363] "A reading means" refers to a speaker or speech synthesis software for outputting the converted text as speech.

[0364] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotions from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotions, and means for reading the converted text aloud.

[0365] Hardware and Software Configuration

[0366] Hardware

[0367] Smart glasses: Equipped with cameras and sensors to capture sign language motions.

[0368] Computer: Built into the smart glasses, it processes data.

[0369] software

[0370] OpenCV: For capturing and pre-processing camera images.

[0371] Keras: Runs AI models for sign language and emotion recognition.

[0372] pyttsx3: Used as a text-to-speech engine.

[0373] Data processing and calculation

[0374] Sign language motion capture

[0375] We use the camera and sensors installed in the smart glasses to capture the user's sign language motion in real time. We use OpenCV to acquire and pre-process the video.

[0376] Sign language motion conversion

[0377] The preprocessed video data is fed into a Keras-based sign language recognition model to convert the sign language motion into text. It is also possible to convert it into another sign language (e.g., from Japanese Sign Language to International Sign Language).

[0378] Emotion recognition

[0379] The same video data is then fed into an emotion recognition model using Keras to analyze the user's emotions. The model recognizes emotions from the speed, strength, and rhythm of the sign language motion.

[0380] Text Adjustment

[0381] Adjust the sign-to-text conversion based on the recognized emotion. For example, if the user is expressing anger, include a statement in the text that reflects that emotion.

[0382] View and read text aloud

[0383] The converted text is displayed on the smart glasses display and then read aloud using pyttsx3.

[0384] Examples

[0385] When a hearing-impaired person uses sign language to ask a store clerk, "Where is this item?", the smart glasses will recognize the sign language and tell the store clerk, "Where is this item?" Furthermore, the emotion engine will recognize that the user is in a hurry based on the speed and strength of the sign language, and add, "You seem to be in a hurry."

[0386] Examples of prompt statements

[0387] The customer signs, asking, "Where is this item?" From the speed and force of the sign, you recognize that the customer is in a hurry, so you verbally tell the store associate, "Where is this item? It seems like you're in a hurry."

[0388] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0389] Step 1:

[0390] A camera mounted on the smart glasses captures the user's sign language motion in real time. The input is the camera image, and the output is the captured image data. The camera image is acquired using OpenCV.

[0391] Step 2:

[0392] The server pre-processes the captured video data. The input is the captured video data and the output is the pre-processed video data. Pre-processing includes video resizing and normalization.

[0393] Step 3:

[0394] The server inputs the preprocessed video data into a sign language recognition model and converts the sign language motions into text. The input is the preprocessed video data and the output is the converted text. The sign language recognition model is implemented using Keras.

[0395] Step 4:

[0396] The server inputs the same preprocessed video data into an emotion recognition model to analyze the user's emotions. The input is the preprocessed video data, and the output is the recognized emotion data. The emotion recognition model is also implemented using Keras.

[0397] Step 5:

[0398] The server adjusts the sign-to-text conversion based on the recognized emotion data. The input is the converted text and the recognized emotion data, and the output is the adjusted text that reflects the emotion. The adjustment includes changing the expression of the text according to the emotion.

[0399] Step 6:

[0400] The server displays the adjusted text on the display of the smart glasses. The input is the adjusted text and the output is the text displayed on the display.

[0401] Step 7:

[0402] The server reads the adjusted text aloud. The input is the adjusted text, and the output is the audio data. The server uses pyttsx3 for the voice reading.

[0403] Example 2

[0404] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] Conventional sign language translation systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. In addition, the translation does not take into account the user's emotions, which leads to a problem of poor communication quality. Furthermore, there is a lack of means for able-bodied people to understand sign language.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0407] In this invention, the server includes a means for capturing sign language motion, a means for analyzing the captured sign language motion, a means for converting the analysis result into another sign language or text, a means for displaying the converted sign language or text, and a means for recognizing the user's emotion and adjusting the conversion result based on the emotion. This makes it possible to accurately analyze the sign language motion and perform translation that takes the user's emotion into account, and also provides a means for able-bodied people to understand sign language.

[0408] "Sign language motion" refers to the movement and position of the hands and fingers when signing.

[0409] "Means of capturing" refers to devices or methods for capturing sign language motion as digital data using cameras, sensors, etc.

[0410] "Means of analysis" refers to devices or methods for analyzing captured sign language motions using AI or machine learning models and understanding their meaning.

[0411] "Means for converting" refers to a device or method that converts sign language motion into another sign language or text based on the analysis results.

[0412] "Means for displaying" refers to a device or method for displaying the converted sign language or text on a user interface.

[0413] "Means for recognizing emotions" refers to a device or method for analyzing emotions from a user's sign language motions and facial expressions and understanding those emotions.

[0414] "Means for adjusting" refers to a device or method that appropriately modifies the conversion result based on the recognized emotion.

[0415] "Japanese Sign Language" refers to a type of sign language used in Japan.

[0416] "International sign language" refers to a type of sign language that is internationally accepted.

[0417] "Text" refers to character information.

[0418] "Normal person" refers to a person without hearing impairment.

[0419] The present invention is a system for capturing and analyzing sign language motion, converting it into another sign language or text, and displaying it. A specific embodiment of this system will be described below.

[0420] First, the user makes a sign language motion toward the camera. The device uses a camera (e.g., a general web camera) to capture the user's sign language motion as video data. This video data is then sent to the server in real time.

[0421] The server sends the received video data to an AI analysis engine (for example, a custom model using TensorFlow). The AI ​​analysis engine analyzes the sign language motion and understands its meaning. For example, it analyzes the motion of "hello" in Japanese sign language and obtains the result.

[0422] Next, the server converts the sign language motion into another sign language (e.g., international sign language) or text based on the analysis results. The converted sign language or text is displayed on a user interface (e.g., a web application built with React.js). The user can check the conversion results on the screen.

[0423] In addition, the system has built-in emotion recognition capabilities. The server uses an emotion recognition engine (e.g., Microsoft Azure® Emotion API) to recognize emotions from the captured sign motion and the user's facial expressions. Based on the recognized emotion, the conversion result can be appropriately adjusted. For example, if the user is expressing an emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[0424] As a concrete example, consider the case where a deaf person makes a motion to say "Hello" in Japanese Sign Language. The device captures that motion with a camera and sends the video data to the server. The server sends the video data to an AI analysis engine and obtains the analysis results. The server then converts the analysis results into "Hello" in international sign language and sends it to the user interface. The device displays the conversion result on the screen. The user can see "Hello" in international sign language on the screen.

[0425] Also, when a healthy person says "Hello," the device captures the voice data with a microphone (e.g., a general USB microphone) and sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). The voice recognition engine converts the utterance into text and displays it on the user interface. The user can see the text "Hello" on the screen.

[0426] An example of a prompt sentence might be:

[0427] "Analyze the sign language motions made by the user facing the camera and convert them from Japanese Sign Language to international sign language. Also, recognize the user's emotions and adjust the conversion results based on those emotions."

[0428] Using this prompt, the generative AI model can analyze the user's sign language motions and emotions and provide appropriate conversion results.

[0429] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0430] Step 1:

[0431] The user makes sign language motions towards the camera.

[0432] As a specific action, the user performs a motion to say "hello" in Japanese sign language. The input is the user's sign language motion, and the output is the video data captured by the camera.

[0433] Step 2:

[0434] The device uses a camera to capture sign language motions.

[0435] In particular, the terminal uses a camera (e.g., a general webcam) to capture the user's sign language motion as video data. The input is the user's sign language motion, and the output is the captured video data.

[0436] Step 3:

[0437] The terminal transmits the captured video data to the server.

[0438] In particular, the terminal transmits captured video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[0439] Step 4:

[0440] The server sends the video data to an AI analysis engine.

[0441] Specifically, the server sends the received video data to an AI analysis engine (e.g., a custom model using TensorFlow). The input is the video data sent to the server, and the output is the video data sent to the AI ​​analysis engine.

[0442] Step 5:

[0443] An AI analysis engine analyzes sign language motions.

[0444] Specifically, the AI ​​analysis engine analyzes video data and recognizes sign language motions. The input is the video data sent to the AI ​​analysis engine, and the output is the analysis results. For example, the motion of "hello" is recognized as Japanese sign language.

[0445] Step 6:

[0446] The server converts the analysis results into another sign language or text.

[0447] Specifically, the server receives the analysis results from the AI ​​analysis engine and converts them into another sign language (for example, international sign language) or text. The input is the analysis result, and the output is the converted sign language or text. For example, "Hello" in Japanese Sign Language is converted into "Hello" in international sign language.

[0448] Step 7:

[0449] The server sends the conversion result to the user interface.

[0450] In particular, the server sends the conversion result to the user interface. The input is the converted sign language or text, and the output is the conversion result sent to the user interface.

[0451] Step 8:

[0452] The terminal displays the conversion result on the user interface.

[0453] Specifically, the terminal displays the conversion result on a user interface (e.g., a web application built with React.js). The input is the conversion result sent to the user interface, and the output is the conversion result displayed on the screen. The user can check the conversion result on the screen.

[0454] (Application example 2)

[0455] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] Conventional sign language translation systems can capture sign language motions and convert them into other sign languages ​​or text, but they lack the ability to recognize the user's emotions and adjust the conversion based on those emotions, which reduces the quality of communication. In addition, there is a lack of appropriate tools to enable smooth communication between hearing-impaired and hearing-speaking people in brick-and-mortar stores.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0458] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion, a means for adjusting the conversion based on the recognized emotion, and a means for being installed in the smart glasses. This enables conversion of sign language or text that reflects the user's emotion, realizing smooth communication between hearing-impaired and hearing-normal people in brick-and-mortar stores.

[0459] "Sign language motion" refers to the movement and position of the hands and fingers when signing.

[0460] "Means of capturing" refers to devices or methods for capturing sign language motion using cameras, sensors, etc.

[0461] "Means for converting" refers to software or algorithms used to convert captured sign language motion into another sign language or text.

[0462] "Display means" refers to a display or screen that visually presents the converted sign language or text to the user.

[0463] "Means of recognizing user emotions" refers to software or algorithms for analyzing emotions from a user's facial expressions and movements.

[0464] "Means for adjusting the conversion based on recognized emotions" refers to software or algorithms that appropriately modify the sign language or text conversion results by taking into account the user's emotions.

[0465] "Means installed on the smart glasses" refers to a method of incorporating an application or software into the smart glasses.

[0466] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion, means for adjusting the conversion based on the recognized emotion, and means for being installed on smart glasses.

[0467] The server uses a camera to capture the sign language motion. The camera captures the sign language motion of the user in real time and sends the video data to the server. The server preprocesses the video data using an image processing library such as OpenCV and inputs it into the sign language recognition model.

[0468] The sign language recognition model is built using deep learning frameworks such as Keras to analyze sign motion and convert it into corresponding text or another sign language, which is then displayed on the smart glasses' display.

[0469] In addition, the server uses facial expression recognition algorithms to recognize the user's emotions, for example by analyzing the user's facial expressions to detect emotions such as happiness or sadness. The emotion engine uses natural language processing libraries such as TextBlob to adjust the conversion results based on the detected emotions.

[0470] The smart glasses are equipped with a display to visually present the converted sign language and text to the user. When the user signs "Hello," the motion is captured by the camera and analyzed by the sign language recognition model. The analysis result is displayed as "Hello" on the smart glasses display. Furthermore, if the user expresses the emotion of joy, the emotion engine reflects that emotion and displays "Hello! Nice to see you!".

[0471] Examples of prompts for a generative AI model include:

[0472] "If the user signs to say 'hello,' analyze that motion and convert it into appropriate text. Or, if the user's emotion is joy, adjust the text to reflect that emotion."

[0473] In this way, a system is provided that enables smooth communication between hearing-impaired and hearing-normal people in brick-and-mortar stores.

[0474] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0475] Step 1:

[0476] The server captures the user's sign language motion in real time using a camera. The input is the camera video data, and the output is the preprocessed video data. Specifically, the server uses OpenCV to split the video data into frames and detect the positions of the hands and fingers.

[0477] Step 2:

[0478] The server inputs the preprocessed video data into a sign language recognition model. The input is the preprocessed video data, and the output is text data corresponding to the sign language motions. Specifically, a deep learning model using Keras analyzes the video data and converts the sign language motions into text.

[0479] Step 3:

[0480] The server transmits the converted text data to the smart glasses. The input is the text data from the sign language recognition model, and the output is the text displayed on the display of the smart glasses. Specifically, the server converts the text data into a format for display on the display of the smart glasses and transmits it.

[0481] Step 4:

[0482] The server analyzes the user's facial expressions from the camera image and recognizes emotions. The input is the camera image data, and the output is the recognized emotion data. Specifically, the server uses a facial expression recognition algorithm to analyze the user's facial expressions and detect emotions such as happiness and sadness.

[0483] Step 5:

[0484] The server adjusts the converted text data based on the recognized emotion data. The input is text data and emotion data, and the output is the adjusted text data. Specifically, the server uses a natural language processing library such as TextBlob to modify the text to reflect the emotion.

[0485] Step 6:

[0486] The server retransmits the adjusted text data to the smart glasses, where the input is the adjusted text data and the output is the adjusted text displayed on the display of the smart glasses. Specifically, the server converts the adjusted text data into a format for displaying it again on the display of the smart glasses and transmits it.

[0487] Step 7:

[0488] The user checks the text displayed on the smart glasses display and communicates. The input is the text displayed on the smart glasses display, and the output is the user's understanding and reaction. Specifically, the user reads the displayed text and responds appropriately.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0492] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0501] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0506] "Example 1"

[0507] The system of the present invention captures sign language motion using means such as cameras and sensors.

[0508] The captured motion data is analyzed by AI and converted into another sign language or text. For example, it is possible to capture the motion of "Hello" in Japanese Sign Language and convert it into "Hello" in international sign language. It is also possible to convert the captured motion into text and display the text "Hello." This allows deaf and hearing people to understand each other's statements even when they are participating in the same meeting.

[0509] "Example 2"

[0510] The system of the present invention is operated through a user interface. The user makes a sign language motion toward the camera. The system captures the motion and analyzes it using AI. The analysis results are converted into another sign language or text and displayed on the user interface. For example, if a deaf person makes a motion to say "Hello" in Japanese Sign Language, the system captures the motion, converts it into "Hello" in international sign language, and displays it on the screen. Also, if a hearing person says "Hello," the system converts the utterance into text and displays it on the screen.

[0511] The process flow of each embodiment will be described below.

[0512] "Example 1"

[0513] Step 1: The user performs a sign language motion using a camera, sensor, or other means.

[0514] Step 2: The system captures the motion data.

[0515] Step 3: The captured motion data is analyzed by AI.

[0516] Step 4: The analysis results are converted into another sign language or text.

[0517] Step 5: The conversion result is displayed to the user.

[0518] "Example 2"

[0519] Step 1: The user performs a sign language motion through the system's user interface.

[0520] Step 2: The system captures the motion data.

[0521] Step 3: The captured motion data is analyzed by AI.

[0522] Step 4: The analysis results are converted into another sign language or text.

[0523] Step 5: The conversion result will be displayed on the user interface.

[0524] Example 1

[0525] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0526] Conventional sign language interpretation systems have difficulty accurately capturing sign language motions and converting them into other sign languages ​​or text. In addition, when deaf and hearing people participate in the same conference, it is difficult for them to understand each other's statements in real time. This has hindered smooth communication.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0528] In this invention, the server includes a means for capturing sign language motion, a means for preprocessing the captured sign language motion, a means for inputting the preprocessed sign language motion into a generative AI model for analysis, a means for converting the analysis result into another sign language or text, and a means for displaying the converted sign language or text. This makes it possible to accurately capture sign language motion and convert it into another sign language or text. In addition, even if a deaf person and a hearing person are participating in the same conference, they can understand each other's remarks in real time.

[0529] A "means for capturing sign language motion" is a device or method for capturing a user's hand movements using a camera or sensor and collecting sign language motion data.

[0530] The "means for pre-processing the captured sign language motion" refers to a device or method that performs processing such as noise removal, data normalization, and frame interpolation on the captured motion data to improve the quality of the data.

[0531] "Means for inputting preprocessed sign language motion into a generative AI model for analysis" refers to a device or method for inputting preprocessed motion data into an AI model and analyzing sign language motion.

[0532] "Means for converting the analysis results into another sign language or text" refers to a device or method for converting the analysis results of an AI model into another sign language or text.

[0533] A "means for displaying the converted sign language or text" is a device or method that visually displays the converted sign language or text to a user.

[0534] "Japanese Sign Language" is a type of sign language used in Japan, a means of communication using specific hand movements and gestures.

[0535] An "international sign language" is a type of sign language used internationally, which is a common means of communication between sign language users in different countries and regions.

[0536] A "generative AI model" is an artificial intelligence model trained to analyze sign language motion data and convert it into other sign languages ​​or text.

[0537] This invention is a system that uses cameras and sensors to capture sign language motions, collects sign language motion data, analyzes it using AI, and converts it into another sign language or text. A specific embodiment of this system will be described below.

[0538] First, the user activates a camera and a sensor (e.g., Microsoft Kinect or Leap Motion) to capture the motion of the sign language, which then captures the user's hand movements in real time.

[0539] The device then transmits the captured sign language motion data to the server, which then pre-processes the received motion data, including noise removal, data normalization, frame interpolation, etc., to improve the quality of the data.

[0540] The preprocessed data is input by the server into a generative AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model analyzes the sign language motion and obtains the analysis result. The analysis result indicates which sign language the sign language motion corresponds to.

[0541] The server converts the analysis results into another sign language (for example, from Japanese Sign Language to an international sign language) or into text. The converted data is sent to the user's device and displayed. Specifically, the text is displayed on the screen, and the sign language motion is reproduced as animation.

[0542] As a concrete example, consider the case of capturing the motion of "Hello" in Japanese Sign Language. A camera or sensor captures the user's hand movements and sends the data to a server. The server uses an AI model to analyze this motion data and converts it into the international sign language "Hello." At the same time, it also converts it into the text "Hello" and displays it on the user's device.

[0543] Example of a prompt to be input to a generative AI model:

[0544] "Please analyze the motion data of "Hello" in Japanese Sign Language and convert it into "Hello" in International Sign Language. Also, please generate "Hello" as text."

[0545] With this system, even if deaf and hearing people are participating in the same conference, they can understand what each other is saying in real time, which will facilitate smooth communication.

[0546] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0547] Step 1:

[0548] A user activates a camera or sensor. A user activates a camera or sensor (e.g., Microsoft Kinect or Leap Motion) to capture sign motion, which captures the user's hand movements in real time. The input is the user's hand movements, and the output is the captured motion data.

[0549] Step 2:

[0550] The device captures the motion data of sign language. The device captures the user's hand movements in real time through the activated camera or sensor. Specifically, it collects hand position and movement data for each frame. The input is the hand movement data from the camera or sensor, and the output is the captured motion data.

[0551] Step 3:

[0552] The device transmits the captured data to the server. The device transmits the captured motion data to the server. This data includes hand position information, motion vectors, etc. The input is the captured motion data, and the output is the data transmitted to the server.

[0553] Step 4:

[0554] The server preprocesses the motion data. The server preprocesses the motion data it receives. Preprocessing includes noise removal, data normalization, frame interpolation, etc. This improves the quality of the data. The input is the motion data sent to the server, and the output is the preprocessed data.

[0555] Step 5:

[0556] The server inputs the preprocessed data into a generative AI model. The server inputs the preprocessed data into an AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model is trained to analyze sign language motion. The input is the preprocessed motion data, and the output is the analysis result of the AI ​​model.

[0557] Step 6:

[0558] The server obtains the analysis results of the AI ​​model. The server obtains the analysis results from the AI ​​model. The analysis results indicate which sign language the sign language motion corresponds to. The input is the analysis results of the AI ​​model, and the output is the analyzed sign language data.

[0559] Step 7:

[0560] The server converts the analysis results into another sign language or text. Based on the analysis results, the server converts into another sign language (for example, from Japanese Sign Language to international sign language) or text. Specifically, it generates motion data and text data for the corresponding sign language. The input is the analyzed sign language data, and the output is the converted sign language or text data.

[0561] Step 8:

[0562] The server sends the conversion result to the terminal. The server sends the converted sign language or text data to the terminal. The input is the converted sign language or text data, and the output is the data sent to the terminal.

[0563] Step 9:

[0564] The terminal displays the conversion result. The terminal displays the received sign language or text data to the user. Specifically, it displays the text on the screen or reproduces the sign language motion with animation. The input is the sign language or text data sent to the terminal, and the output is the information displayed to the user.

[0565] (Application example 1)

[0566] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0567] There is a problem that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult for hearing-impaired people to communicate with hearing-nonhearing people who cannot understand sign language, which often causes inconvenience when serving customers or giving directions. In addition, there is a lack of technology to convert sign language into voice or text, making real-time communication difficult.

[0568] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0569] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for analyzing the captured sign language motion and converting it into voice, and a means for converting voice into text or sign language and displaying it. This enables hearing-impaired people and hearing-normal people to communicate smoothly in a brick-and-mortar store.

[0570] A "means for capturing sign language motion" is a device or technology that uses a camera or sensor to capture sign language movements.

[0571] "Means for converting captured sign language motion into another sign language or text" refers to technology that uses AI or machine learning models to convert captured sign language motion into another sign language format or text.

[0572] A "means for displaying converted signs or text" is a device or technology that uses a display or screen to visually present the converted signs or text.

[0573] "Means of analyzing captured sign language motion and converting it into audio" refers to technology that uses AI and voice synthesis technology to convert captured sign language movements into audio.

[0574] "Means for converting speech into text or sign language and displaying it" refers to a device or technology that uses speech recognition technology to convert speech into text or sign language and display it visually.

[0575] A system for implementing this invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for analyzing the captured sign language motion and converting it into speech, and means for converting speech into text or sign language and displaying it.

[0576] 1. System Programming

[0577] The program for this system is implemented using Python. Specifically, it uses OpenCV to capture sign language motion from the camera, loads the AI ​​model using Keras, and analyzes the sign language motion and converts it into text and audio.

[0578] 2. Program Processing

[0579] The server captures the sign language motion using a camera (e.g. Logitech C920). The captured video is pre-processed using OpenCV and resized to 64x64 pixels. It is then input to an AI model loaded using Keras to analyze the sign language motion. The analysis results are converted into text or audio and displayed or played through a display or speaker.

[0580] 3. Specific Examples

[0581] For example, consider a customer service scenario in a brick-and-mortar store. When a hearing-impaired person wears smart glasses and says "hello" in sign language, the camera captures their motion, and the AI ​​analyzes it to display the text "hello" on the smart glasses' display. Conversely, when a store clerk says "Welcome," the speech is converted into text and displayed as sign language.

[0582] 4. Examples of prompts

[0583] Here is an example of a prompt to be input to the generative AI model:

[0584] "I want to develop a sign language interpretation application. I want to create a system that captures sign language motion with a camera, analyzes it with AI, and converts it into text. Can you please tell me a program that uses Python, OpenCV, and Keras to capture sign language motion and analyze it with an AI model?"

[0585] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0586] Step 1:

[0587] The server captures the sign language motion using a camera. The input is video data from the camera, and the output is the captured sign language motion video. Specifically, the camera captures video in real time and transmits the video to the server frame by frame.

[0588] Step 2:

[0589] The server preprocesses the captured video using OpenCV. The input is the captured sign language motion video, and the output is the preprocessed video data. Specifically, the video is resized to 64x64 pixels and converted into a format that can be input to the AI ​​model.

[0590] Step 3:

[0591] The server inputs the preprocessed video data into the AI ​​model loaded using Keras. The input is the preprocessed video data, and the output is the sign language motion analysis result. In concrete terms, the AI ​​model analyzes the video data and generates prediction results for converting the sign language motion into text and audio.

[0592] Step 4:

[0593] The server converts the analysis results into text or audio. The input is the prediction result of the AI ​​model, and the output is text data or audio data. Specifically, it generates text based on the prediction result, and generates audio using speech synthesis technology if necessary.

[0594] Step 5:

[0595] The server displays or plays the generated text or audio through a display or speaker. The input is text data or audio data, and the output is visual or auditory information. As a specific operation, the text is displayed on the display and the audio is played from the speaker.

[0596] Step 6:

[0597] The user sees the text shown on the display or the audio played from the speaker. The input is the visual or auditory information, and the output is the user's understanding. In concrete terms, the user reads the displayed text or listens to the played audio.

[0598] Example 2

[0599] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0600] Conventional sign language recognition systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. They also lacked the functionality to convert speech into text and display it in a format that can be read by hearing-impaired people. This has led to problems with smooth communication between hearing-impaired people and hearing-able people.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0602] In this invention, the server includes means for capturing sign language motion, means for analyzing the captured sign language motion, means for converting the analyzed sign language motion into another sign language or text, means for displaying the converted sign language or text, means for capturing voice, means for converting the captured voice into text, and means for displaying the converted text. This makes it possible to accurately analyze sign language motion and convert it into another sign language or text, and further convert voice into text and display it, enabling smooth communication between hearing-impaired people and hearing-normal people.

[0603] The "means for capturing sign language motion" refers to a device or method for capturing the sign language movements made by a user as video data using a photographing device such as a camera.

[0604] "Means for analyzing the captured sign language motion" refers to an AI model or algorithm for analyzing the captured video data and identifying the content of the sign language.

[0605] "Means for converting analyzed sign language motion into another sign language or text" refers to a device or method for converting the sign language content into another sign language or text format based on the analysis results.

[0606] A "means for displaying the converted sign language or text" is a display or software for displaying the converted sign language or text in a user interface.

[0607] The "means for capturing voice" refers to a device or method for capturing a user's speech as voice data using a voice input device such as a microphone.

[0608] "Means for converting captured speech into text" refers to speech recognition software or algorithms used to analyze the captured speech data and convert it into text form.

[0609] "Means for displaying the converted text" refers to a display or software for displaying the converted text on a user interface.

[0610] This invention is a system that captures sign language motion, analyzes it, and converts it into another sign language or text. It also includes a function to capture voice, convert it into text, and display it. A specific embodiment of this system will be described below.

[0611] Hardware and Software Used

[0612] Hardware:

[0613] Camera: A high-resolution camera is used to capture the user's sign language motions.

[0614] Microphone: An audio input device for capturing the user's voice.

[0615] Display: A screen for displaying the user interface.

[0616] software:

[0617] AI model for sign language recognition: Analyzes sign language motions using machine learning frameworks such as TensorFlow and PyTorch.

[0618] Speech Recognition Software: Convert speech to text using speech recognition technology such as Google Speech-to-Text.

[0619] System Operation

[0620] Sign Language Motion Capture and Analysis:

[0621] When a user performs a sign language motion in front of the camera, the device captures the motion. The captured video data is sent to a server, which analyzes the video data using an AI model for sign language recognition. The analysis results are converted into another sign language or text and sent to the device. The device displays the conversion results on the user interface.

[0622] Audio capture and conversion:

[0623] When a user speaks, the device captures the speech using a microphone. The captured speech data is sent to a server, which uses voice recognition software to convert the speech into text. The converted text is sent to the device and displayed on a user interface.

[0624] Examples

[0625] Sign language motion capture and conversion:

[0626] The user makes the motion to say "hello" in Japanese sign language in front of the camera.

[0627] The device captures the motion using a camera and transmits the video data to a server.

[0628] The server inputs the video data into an AI model for sign language recognition and analyzes it.

[0629] The server converts the analysis result into the international sign language "hello" and sends it to the terminal.

[0630] The terminal displays the conversion result on the user interface.

[0631] Audio capture and conversion:

[0632] The user says "Hello."

[0633] The terminal captures the audio using a microphone and transmits the audio data to the server.

[0634] The server inputs the voice data into speech recognition software and converts it into text.

[0635] The server transmits the conversion result to the terminal.

[0636] The terminal displays the conversion result on the user interface.

[0637] Examples of prompt statements

[0638] "If I make a motion to say 'hello' in Japanese sign language, please convert it into international sign language."

[0639] "If you say 'hello' by voice, convert that speech to text."

[0640] In this way, the server and the terminal work together to realize a system that analyzes the user's sign language and voice, converts it into an appropriate format, and displays it.

[0641] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0642] Step 1:

[0643] The user makes sign language motions towards the camera.

[0644] As a specific action, a user makes a motion to say "hello" in Japanese sign language in front of the camera. The input is the user's sign language motion, and the output is the video data captured by the camera.

[0645] Step 2:

[0646] The device uses a camera to capture sign language motions.

[0647] Specifically, the device captures the user's sign language motion as video data using a high-resolution camera. The input is the user's sign language motion, and the output is the captured video data.

[0648] Step 3:

[0649] The terminal transmits the captured video data to the server.

[0650] In particular, the terminal transmits video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[0651] Step 4:

[0652] The server inputs the video data into an AI model for sign language recognition.

[0653] Specifically, the server inputs the received video data into an AI model for sign language recognition (e.g., TensorFlow or PyTorch). The input is the video data sent to the server, and the output is the data input into the AI ​​model.

[0654] Step 5:

[0655] The server uses an AI model to analyze sign language motions.

[0656] Specifically, the server uses the AI ​​model to analyze the video data and identify the sign language content. The input is the video data input to the AI ​​model, and the output is the analyzed sign language content.

[0657] Step 6:

[0658] The server converts the analysis results into another sign language or text.

[0659] Specifically, the server converts the sign language content into another sign language (e.g., international sign language) or text format based on the analysis results. The input is the analyzed sign language content, and the output is the converted sign language or text.

[0660] Step 7:

[0661] The server transmits the conversion result to the terminal.

[0662] Specifically, the server sends the conversion result to the terminal. The input is the converted sign language or text, and the output is the conversion result sent to the terminal.

[0663] Step 8:

[0664] The terminal displays the conversion result on the user interface.

[0665] As a specific operation, the terminal displays the conversion result on a user interface. The input is the conversion result sent to the terminal, and the output is the sign language or text displayed on the user interface.

[0666] Step 9:

[0667] The user speaks aloud.

[0668] As a specific operation, the user says "Hello." The input is the user's voice, and the output is the voice data captured by the microphone.

[0669] Step 10:

[0670] The device captures audio using a microphone.

[0671] In particular, the terminal uses a microphone to capture the user's voice as voice data, with the input being the user's voice and the output being the captured voice data.

[0672] Step 11:

[0673] The device transmits the captured audio data to the server.

[0674] In particular, the terminal transmits voice data to the server in real time, where the input is the captured voice data and the output is the voice data transmitted to the server.

[0675] Step 12:

[0676] The server inputs the voice data into voice recognition software.

[0677] Specifically, the server inputs the received voice data into speech recognition software (e.g. Google Speech-to-Text). The input is the voice data sent to the server, and the output is the data input into the speech recognition software.

[0678] Step 13:

[0679] The server converts the voice data into text.

[0680] Specifically, the server uses speech recognition software to convert the voice data into text, with the input being the voice data entered into the speech recognition software and the output being the converted text.

[0681] Step 14:

[0682] The server transmits the conversion result to the terminal.

[0683] In concrete operation, the server sends the conversion result to the terminal, where the input is the converted text and the output is the conversion result sent to the terminal.

[0684] Step 15:

[0685] The terminal displays the conversion result on the user interface.

[0686] In particular, the terminal displays the conversion result on a user interface, where the input is the conversion result sent to the terminal and the output is the text displayed on the user interface.

[0687] (Application example 2)

[0688] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0689] Conventional sign language translation systems have limited functionality for converting sign language into text and translating between different sign languages, making it difficult for deaf and hearing people to communicate smoothly. In addition, there is a lack of means for deaf and hearing people to communicate smoothly in physical stores.

[0690] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0691] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for converting voice into text, a means for displaying the converted text, and a means for translating between different sign languages, thereby enabling deaf and hearing people to communicate smoothly in a brick-and-mortar store.

[0692] A "sign language motion capture means" is a device or method that uses a camera or sensor to capture the sign language movements made by a user.

[0693] A "means for converting captured sign language motion into another sign language or text format" is software or algorithms that analyzes captured sign language motions and converts them into another sign language or text format.

[0694] A "means for displaying the converted sign language or text" is a display or screen that visually displays the converted sign language or text in a user interface.

[0695] A "speech-to-text conversion means" is software or algorithms used to convert speech data into text data using speech recognition technology.

[0696] An "interlanguage translation tool" is a translation algorithm or software that converts from one sign language to another.

[0697] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for converting speech to text, means for displaying the converted text, and means for translating between different sign languages.

[0698] System Programming

[0699] The system program performs the following processing.

[0700] 1. Means of capturing sign language motion:

[0701] The system uses cameras and sensors to capture the sign language movements made by the user, specifically, cameras installed in smartphones and smart glasses.

[0702] 2. How to convert captured sign motion into other signs or text:

[0703] It uses machine learning libraries such as TensorFlow to analyze the captured sign language movements and convert them into other sign languages ​​or text formats, using pre-trained generative AI models.

[0704] 3. Means for displaying converted signs and / or text:

[0705] Use the display of a smartphone or smart glasses to visually display the converted sign language or text on a user interface.

[0706] 4. Ways to convert speech to text:

[0707] The SpeechRecognition library is used to convert voice data into text data using speech recognition technology, specifically, using the microphones on smartphones and smart glasses.

[0708] 5. Means of translation between different sign languages:

[0709] Using translation algorithms or software to convert from one sign language to another, including using generative AI models to translate.

[0710] Processing flow

[0711] The server takes in video from the camera and captures the sign language motion. The captured video is analyzed using TensorFlow and converted into another sign language or text. The converted sign language or text is displayed on the display of the smartphone or smart glasses. Using voice recognition technology, the voice data is converted into text data, which is also displayed. Translation between different sign languages ​​is also performed using generative AI models.

[0712] Examples

[0713] For example, if a deaf customer says "Hello" in sign language, the camera in the smart glasses will capture that motion, and the AI ​​will convert it into text and display it on the store clerk's smartphone. If the store clerk says "Welcome," the speech will be converted into text and displayed on the deaf customer's smart glasses.

[0714] Examples of prompt statements

[0715] "When a user says 'Hello' in sign language, capture that motion with a camera, analyze it with an AI model, and convert it into text as 'Hello.' Also, when a store clerk says 'Welcome,' convert that speech into text and display it."

[0716] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0717] Step 1:

[0718] A user performs a sign language motion. The camera in a smartphone or smart glasses captures the motion. The input is the user's sign language motion, and the output is the captured video data. In concrete terms, the camera continuously shoots video and captures the sign language motion in real time.

[0719] Step 2:

[0720] The server receives the captured video data and performs preprocessing. The input is the captured video data, and the output is the preprocessed image data. The specific operations include resizing the video data, removing noise, and extracting the sign language motion features.

[0721] Step 3:

[0722] The server inputs the preprocessed image data into the generative AI model to analyze the sign language motion. The input is the preprocessed image data, and the output is the analysis result, a text representation of the sign language. In concrete terms, the generative AI model analyzes the characteristics of the sign language motion and generates the corresponding text.

[0723] Step 4:

[0724] The server converts the results of the analysis into another sign language or text. The input is a textual representation of the sign language, and the output is the converted sign language or text. In particular, a translation algorithm converts the textual representation of the sign language into a different sign language or text.

[0725] Step 5:

[0726] The server sends the converted sign language or text to the terminal and displays it. The input is the converted sign language or text, and the output is the information displayed on the terminal's display. The specific operation is to visually display the converted sign language or text on the terminal's display.

[0727] Step 6:

[0728] A user speaks using voice. The microphone in a smartphone or smart glasses captures the voice. The input is the user's voice, and the output is the captured voice data. In concrete terms, the microphone continuously records the voice and captures the voice data in real time.

[0729] Step 7:

[0730] The server receives the captured voice data and performs speech recognition. The input is the captured voice data, and the output is text data that is the result of speech recognition. In concrete terms, the speech recognition algorithm analyzes the voice data and generates corresponding text.

[0731] Step 8:

[0732] The server sends the speech recognition results to the terminal and displays them. The input is the text data of the speech recognition results, and the output is the information displayed on the terminal's display. The specific operation is to visually display the speech recognition results on the terminal's display.

[0733] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0734] "Example 1"

[0735] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes the user's emotion from the captured sign motion. Specifically, the emotion engine determines the likelihood that the user is expressing an emotion such as joy, anger, sadness, surprise, etc. from the speed, strength, and rhythm of the sign motion. The emotion engine adjusts the sign-to-text conversion according to the emotion. For example, if it is determined that the user is expressing an emotion of anger, an expression reflecting that emotion can be included in the text.

[0736] "Example 2"

[0737] In another embodiment of the present invention, an emotion engine is incorporated to recognize the emotion of the user from the captured sign motion and adjust the conversion from Japanese Sign Language to International Sign Language based on the emotion. Specifically, if it is determined that the user is expressing an emotion of joy, an expression reflecting the emotion can be included in the international sign language.

[0738] The process flow of each embodiment will be described below.

[0739] "Example 1"

[0740] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[0741] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[0742] Step 3: The emotion engine adjusts the sign-to-text conversion based on the determined emotion. For example, if it determines that the user is expressing an emotion of anger, it can include a wording that reflects that emotion in the text.

[0743] "Example 2"

[0744] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[0745] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[0746] Step 3: The emotion engine adjusts the conversion from Japanese Sign Language to international sign language based on the determined emotion. For example, if it determines that the user is expressing the emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[0747] Example 1

[0748] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0749] Conventional sign language translation systems were able to capture sign language motions and convert them into other sign languages ​​or text, but they lacked the ability to recognize the user's emotions and adjust the translation results accordingly. This made it difficult to accurately convey the nuances and emotions of sign language, resulting in a decline in the quality of communication.

[0750] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0751] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion from the captured sign language motion, and a means for adjusting the conversion from sign language to text in accordance with the recognized emotion. This enables sign language translation that reflects the user's emotion, realizing more natural and accurate communication.

[0752] A "means for capturing sign language motion" is a device or method that uses a camera or motion sensor to capture a user's sign language movements in real time and acquire them as digital data.

[0753] "Means for converting captured sign language motion into another sign language or text" refers to a device or method that uses an AI analysis engine to analyze acquired sign language motion data and convert it into a corresponding other sign language or text.

[0754] "Means for displaying the converted sign language or text" refers to a device or method, such as a display or monitor, for visually presenting the converted sign language or text to a user.

[0755] The "means for recognizing a user's emotions from the captured sign language motion" refers to an emotion engine or algorithm that analyzes characteristics of the sign language motion, such as speed, strength, and rhythm, and estimates the user's emotions.

[0756] "Means for adjusting the sign language-to-text conversion in response to recognized emotions" refers to an apparatus or method that adjusts the sign language-to-text conversion result based on the user's emotions recognized by an emotion engine, and generates text that reflects the emotions.

[0757] The system of the present invention includes a camera and a motion sensor for capturing the motion of the sign language, an AI analysis engine for analyzing the captured motion data, an emotion engine for recognizing the emotion of the user, and a display for displaying the converted sign language and text. Specific embodiments are described below.

[0758] Hardware and Software Used

[0759] Hardware: High resolution camera, motion sensor, display

[0760] Software: AI analysis engine, emotion engine, sign language conversion algorithm

[0761] Data Import

[0762] When a user signs, the device's camera and motion sensor capture the sign language motion in real time. The camera captures hand movements in high resolution, and the motion sensor records the hand position and movement in detail. The captured motion data is then sent from the device to a server.

[0763] 4. Data Analysis

[0764] The server inputs the received motion data into an AI analysis engine, which uses a deep learning model to analyze the motion data and convert it into the corresponding sign language or text, for example converting "hello" in Japanese sign language to "hello" in international sign language.

[0765] emotion recognition

[0766] The server inputs the analyzed sign language data into the emotion engine. The emotion engine extracts features of the sign language motion, such as speed, strength, and rhythm, to recognize the user's emotion. For example, if the sign language is fast and strong, it determines that the user is expressing the emotion of anger. It adjusts the conversion from sign language to text according to the recognized emotion. For example, if the user is expressing the emotion of joy, it generates the text "Hello!" that reflects that emotion.

[0767] Data Output

[0768] The server sends the converted sign language or text to the terminal, which displays the received data to the user. For example, the text "Hello!" is displayed on the terminal screen.

[0769] Examples

[0770] For example, when a user wants to say "hello" in Japanese Sign Language, the following specific actions are performed:

[0771] 1. The user signs.

[0772] 2. The device's camera and motion sensors capture the user's sign language motions in real time.

[0773] 3. The device transmits the captured motion data to the server.

[0774] 4. The server inputs the received motion data into the AI ​​analysis engine.

[0775] 5. The AI ​​analysis engine analyzes the motion data and converts the Japanese sign language "hello" into the international sign language "hello."

[0776] 6. The server inputs the analyzed sign language data into the emotion engine.

[0777] 7. The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[0778] 8. The emotion engine determines that the user is expressing the emotion of happiness and generates text that reflects that emotion: "Hello!"

[0779] 9. The server sends the converted sign language or text to the terminal.

[0780] 10. The device displays the received data to the user, for example displaying the text "Hello!" on the device screen.

[0781] Examples of prompt statements

[0782] Examples of prompt sentences to be input to a generative AI model include the following:

[0783] "Analyze the sign language motion data captured by the camera and convert the Japanese sign language "Hello" into the international sign language "Hello." Also, recognize the user's emotions, and if the emotion includes joy, display "Hello!" in the text."

[0784] In this way, the system can analyze sign language motions in real time and convert them into appropriate sign language and text.

[0785] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0786] Step 1:

[0787] The user signs.

[0788] As a specific action, the user says "hello" in Japanese Sign Language.

[0789] Input: User's sign language motion

[0790] Output: None

[0791] Step 2:

[0792] The device's camera and motion sensors capture the user's sign language movements in real time.

[0793] Specifically, the camera captures hand movements in high resolution, while the motion sensor records the hand's position and movement in detail.

[0794] Input: User's sign language motion

[0795] Output: Motion data

[0796] Step 3:

[0797] The terminal transmits the captured motion data to the server.

[0798] Specifically, data is transmitted in real time, minimizing latency.

[0799] Input: Motion data

[0800] Output: Motion data sent to the server

[0801] Step 4:

[0802] The server inputs the received motion data into an AI analysis engine.

[0803] Specifically, the AI ​​analytics engine uses deep learning models to analyze motion data.

[0804] Input: Motion data sent to the server

[0805] Output: Analysis results (sign language and text)

[0806] Step 5:

[0807] The server inputs the parsed sign language data into an emotion engine.

[0808] As a specific action, the emotion engine extracts features such as speed, strength and rhythm of sign language motion.

[0809] Input: Analysis results (sign language or text)

[0810] Output: Emotion data

[0811] Step 6:

[0812] The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[0813] Specifically, if the sign language is fast and forceful, it is determined that the user is expressing anger.

[0814] Input: Emotion data

[0815] Output: Recognized emotion

[0816] Step 7:

[0817] The emotion engine adjusts the sign language to text conversion depending on the recognized emotion.

[0818] Specifically, if the user expresses an emotion of joy, the system generates text that reflects that emotion: "Hello!"

[0819] Input: Recognized emotion

[0820] Output: Adjusted text

[0821] Step 8:

[0822] The server sends the converted sign language or text to the terminal.

[0823] Specifically, the data is encrypted and transmitted securely.

[0824] Input: Adjusted text

[0825] Output: Text sent to the terminal.

[0826] Step 9:

[0827] The terminal displays the received data to the user.

[0828] The specific action is that the text "Hello!" will be displayed on the device screen.

[0829] Input: Text sent to the terminal

[0830] Output: The displayed text

[0831] (Application example 1)

[0832] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0833] There is an issue that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult to convey information using sign language to hearing-nonhearing people who cannot understand sign language. In addition, there is a lack of technology that can recognize the user's emotions from the motion of sign language and convert it into text or voice that appropriately reflects those emotions, which makes it difficult to communicate accurately with emotions.

[0834] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0835] In this invention, the server includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotion, and means for reading out the converted text aloud. This enables hearing-impaired people and hearing-normal people to communicate smoothly in physical stores.

[0836] "Sign language motion" refers to the movements of the hands, arms, fingers, etc. when using sign language.

[0837] "Means of capturing" refers to devices or methods that use cameras, sensors, etc. to capture sign language motion as digital data.

[0838] A "conversion method" refers to software or algorithms used to convert captured sign motion into another sign language or text.

[0839] "Display device" means a display or screen for visually presenting the converted sign language or text.

[0840] "Means of recognizing emotions" refers to software or algorithms for analyzing and recognizing a user's emotions from sign language motions.

[0841] "Means for tailoring the conversion" means software or algorithms that appropriately adjust the conversion of sign language to text based on recognized emotions.

[0842] "A reading means" refers to a speaker or speech synthesis software for outputting the converted text as speech.

[0843] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotions from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotions, and means for reading aloud the converted text.

[0844] Hardware and Software Configuration

[0845] Hardware

[0846] Smart glasses: Equipped with cameras and sensors to capture sign language motions.

[0847] Computer: Built into the smart glasses, it processes data.

[0848] software

[0849] OpenCV: For capturing and pre-processing camera images.

[0850] Keras: Runs AI models for sign language and emotion recognition.

[0851] pyttsx3: Used as a text-to-speech engine.

[0852] Data processing and calculation

[0853] Sign language motion capture

[0854] We use the camera and sensors installed in the smart glasses to capture the user's sign language motion in real time. We use OpenCV to acquire and pre-process the video.

[0855] Sign language motion conversion

[0856] The preprocessed video data is fed into a Keras-based sign language recognition model to convert the sign language motion into text. It is also possible to convert it into another sign language (e.g., from Japanese Sign Language to International Sign Language).

[0857] Emotion recognition

[0858] The same video data is then fed into an emotion recognition model using Keras to analyze the user's emotions. The model recognizes emotions from the speed, strength, and rhythm of the sign language motion.

[0859] Text Adjustment

[0860] Adjust the sign-to-text conversion based on the recognized emotion. For example, if the user is expressing anger, include a statement in the text that reflects that emotion.

[0861] View and read text aloud

[0862] The converted text is displayed on the smart glasses display and then read aloud using pyttsx3.

[0863] Examples

[0864] When a hearing-impaired person uses sign language to ask a store clerk, "Where is this item?", the smart glasses will recognize the sign language and tell the store clerk, "Where is this item?" Furthermore, the emotion engine will recognize that the user is in a hurry based on the speed and strength of the sign language, and add, "You seem to be in a hurry."

[0865] Examples of prompt statements

[0866] The customer signs, "Where is this item?". From the speed and force of the sign, you recognize that the customer is in a hurry, so you verbally tell the store associate, "Where is this item? It seems like you're in a hurry."

[0867] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0868] Step 1:

[0869] A camera mounted on the smart glasses captures the user's sign language motion in real time. The input is the camera image, and the output is the captured image data. The camera image is acquired using OpenCV.

[0870] Step 2:

[0871] The server pre-processes the captured video data. The input is the captured video data and the output is the pre-processed video data. Pre-processing includes video resizing and normalization.

[0872] Step 3:

[0873] The server inputs the preprocessed video data into a sign language recognition model and converts the sign language motions into text. The input is the preprocessed video data and the output is the converted text. The sign language recognition model is implemented using Keras.

[0874] Step 4:

[0875] The server inputs the same preprocessed video data into an emotion recognition model to analyze the user's emotions. The input is the preprocessed video data, and the output is the recognized emotion data. The emotion recognition model is also implemented using Keras.

[0876] Step 5:

[0877] The server adjusts the sign-to-text conversion based on the recognized emotion data. The input is the converted text and the recognized emotion data, and the output is the adjusted text that reflects the emotion. The adjustment includes changing the expression of the text according to the emotion.

[0878] Step 6:

[0879] The server displays the adjusted text on the display of the smart glasses. The input is the adjusted text and the output is the text displayed on the display.

[0880] Step 7:

[0881] The server reads the adjusted text aloud. The input is the adjusted text, and the output is the audio data. The server uses pyttsx3 for the voice reading.

[0882] Example 2

[0883] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0884] Conventional sign language translation systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. In addition, the translation does not take into account the user's emotions, which leads to a problem of poor communication quality. Furthermore, there is a lack of means for able-bodied people to understand sign language.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0886] In this invention, the server includes a means for capturing sign language motion, a means for analyzing the captured sign language motion, a means for converting the analysis result into another sign language or text, a means for displaying the converted sign language or text, and a means for recognizing the user's emotion and adjusting the conversion result based on the emotion. This makes it possible to accurately analyze the sign language motion and perform translation that takes the user's emotion into account, and also provides a means for able-bodied people to understand sign language.

[0887] "Sign language motion" refers to the movement and position of the hands and fingers when signing.

[0888] "Means of capturing" refers to devices or methods for capturing sign language motion as digital data using cameras, sensors, etc.

[0889] "Means of analysis" refers to devices or methods for analyzing captured sign language motions using AI or machine learning models and understanding their meaning.

[0890] "Means for converting" refers to a device or method that converts sign language motion into another sign language or text based on the analysis results.

[0891] "Means for displaying" refers to a device or method for displaying the converted sign language or text on a user interface.

[0892] "Means for recognizing emotions" refers to a device or method for analyzing emotions from a user's sign language motions and facial expressions and understanding those emotions.

[0893] "Means for adjusting" refers to a device or method that appropriately modifies the conversion result based on the recognized emotion.

[0894] "Japanese Sign Language" refers to a type of sign language used in Japan.

[0895] "International sign language" refers to a type of sign language that is internationally accepted.

[0896] "Text" refers to character information.

[0897] "Normal person" refers to a person without hearing impairment.

[0898] The present invention is a system for capturing and analyzing sign language motion, converting it into another sign language or text, and displaying it. A specific embodiment of this system will be described below.

[0899] First, the user makes a sign language motion toward the camera. The device uses a camera (e.g., a general web camera) to capture the user's sign language motion as video data. This video data is then sent to the server in real time.

[0900] The server sends the received video data to an AI analysis engine (for example, a custom model using TensorFlow). The AI ​​analysis engine analyzes the sign language motion and understands its meaning. For example, it analyzes the motion of "hello" in Japanese sign language and obtains the result.

[0901] Next, the server converts the sign language motion into another sign language (e.g., international sign language) or text based on the analysis results. The converted sign language or text is displayed on a user interface (e.g., a web application built with React.js). The user can check the conversion results on the screen.

[0902] In addition, the system has built-in emotion recognition capabilities. The server uses an emotion recognition engine (e.g., Microsoft Azure Emotion API) to recognize emotions from the captured sign motion and the user's facial expressions. Based on the recognized emotion, the conversion result can be adjusted appropriately. For example, if the user is expressing the emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[0903] As a concrete example, consider the case where a deaf person makes a motion to say "Hello" in Japanese Sign Language. The device captures that motion with a camera and sends the video data to the server. The server sends the video data to an AI analysis engine and obtains the analysis results. The server then converts the analysis results into "Hello" in international sign language and sends it to the user interface. The device displays the conversion result on the screen. The user can see "Hello" in international sign language on the screen.

[0904] Also, when a healthy person says "Hello," the device captures the voice data with a microphone (e.g., a general USB microphone) and sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). The voice recognition engine converts the utterance into text and displays it on the user interface. The user can see the text "Hello" on the screen.

[0905] An example of a prompt sentence might be:

[0906] "Analyze the sign language motions made by the user facing the camera and convert them from Japanese Sign Language to international sign language. Also, recognize the user's emotions and adjust the conversion results based on those emotions."

[0907] Using this prompt, the generative AI model can analyze the user's sign language motions and emotions and provide appropriate conversion results.

[0908] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0909] Step 1:

[0910] The user makes sign language motions towards the camera.

[0911] As a specific action, the user performs a motion to say "hello" in Japanese sign language. The input is the user's sign language motion, and the output is the video data captured by the camera.

[0912] Step 2:

[0913] The device uses a camera to capture sign language motions.

[0914] In particular, the terminal uses a camera (e.g., a general webcam) to capture the user's sign language motion as video data. The input is the user's sign language motion, and the output is the captured video data.

[0915] Step 3:

[0916] The terminal transmits the captured video data to the server.

[0917] In particular, the terminal transmits captured video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[0918] Step 4:

[0919] The server sends the video data to an AI analysis engine.

[0920] Specifically, the server sends the received video data to an AI analysis engine (e.g., a custom model using TensorFlow). The input is the video data sent to the server, and the output is the video data sent to the AI ​​analysis engine.

[0921] Step 5:

[0922] An AI analysis engine analyzes sign language motions.

[0923] Specifically, the AI ​​analysis engine analyzes video data and recognizes sign language motions. The input is the video data sent to the AI ​​analysis engine, and the output is the analysis results. For example, the motion of "hello" is recognized as Japanese sign language.

[0924] Step 6:

[0925] The server converts the analysis results into another sign language or text.

[0926] Specifically, the server receives the analysis results from the AI ​​analysis engine and converts them into another sign language (for example, international sign language) or text. The input is the analysis result, and the output is the converted sign language or text. For example, "Hello" in Japanese Sign Language is converted into "Hello" in international sign language.

[0927] Step 7:

[0928] The server sends the conversion result to the user interface.

[0929] In particular, the server sends the conversion result to the user interface. The input is the converted sign language or text, and the output is the conversion result sent to the user interface.

[0930] Step 8:

[0931] The terminal displays the conversion result on the user interface.

[0932] Specifically, the terminal displays the conversion result on a user interface (e.g., a web application built with React.js). The input is the conversion result sent to the user interface, and the output is the conversion result displayed on the screen. The user can check the conversion result on the screen.

[0933] (Application example 2)

[0934] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0935] Conventional sign language translation systems can capture sign language motions and convert them into other sign languages ​​or text, but they lack the ability to recognize the user's emotions and adjust the conversion based on those emotions, which reduces the quality of communication. In addition, there is a lack of appropriate tools to enable smooth communication between hearing-impaired and hearing-speaking people in brick-and-mortar stores.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0937] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion, a means for adjusting the conversion based on the recognized emotion, and a means for being installed in the smart glasses. This enables conversion of sign language or text that reflects the user's emotion, realizing smooth communication between hearing-impaired and hearing-normal people in brick-and-mortar stores.

[0938] "Sign language motion" refers to the movement and position of the hands and fingers when signing.

[0939] "Means of capturing" refers to devices or methods for capturing sign language motion using cameras, sensors, etc.

[0940] "Means for converting" refers to software or algorithms used to convert captured sign language motion into another sign language or text.

[0941] "Display means" refers to a display or screen that visually presents the converted sign language or text to the user.

[0942] "Means of recognizing user emotions" refers to software or algorithms for analyzing emotions from the user's facial expressions and movements.

[0943] "Means for adjusting the conversion based on recognized emotions" refers to software or algorithms that appropriately modify the sign language or text conversion results by taking into account the user's emotions.

[0944] "Means installed on the smart glasses" refers to a method of incorporating an application or software into the smart glasses.

[0945] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion, means for adjusting the conversion based on the recognized emotion, and means for being installed on smart glasses.

[0946] The server uses a camera to capture the sign language motion. The camera captures the sign language motion of the user in real time and sends the video data to the server. The server preprocesses the video data using an image processing library such as OpenCV and inputs it into the sign language recognition model.

[0947] The sign language recognition model is built using deep learning frameworks such as Keras to analyze sign motion and convert it into corresponding text or another sign language, which is then displayed on the smart glasses' display.

[0948] In addition, the server uses facial expression recognition algorithms to recognize the user's emotions, for example by analyzing the user's facial expressions to detect emotions such as happiness or sadness. The emotion engine uses natural language processing libraries such as TextBlob to adjust the conversion results based on the detected emotions.

[0949] The smart glasses are equipped with a display to visually present the converted sign language and text to the user. When the user signs "Hello," the motion is captured by the camera and analyzed by the sign language recognition model. The analysis result is displayed as "Hello" on the smart glasses display. Furthermore, if the user expresses the emotion of joy, the emotion engine reflects that emotion and displays "Hello! Nice to see you!".

[0950] Examples of prompts for a generative AI model include:

[0951] "If the user signs to say 'hello,' analyze that motion and convert it into appropriate text. Or, if the user's emotion is joy, adjust the text to reflect that emotion."

[0952] In this way, a system is provided that enables smooth communication between hearing-impaired and hearing-normal people in brick-and-mortar stores.

[0953] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0954] Step 1:

[0955] The server captures the user's sign language motion in real time using a camera. The input is the camera video data, and the output is the preprocessed video data. Specifically, the server uses OpenCV to split the video data into frames and detect the positions of the hands and fingers.

[0956] Step 2:

[0957] The server inputs the preprocessed video data into a sign language recognition model. The input is the preprocessed video data, and the output is text data corresponding to the sign language motions. Specifically, a deep learning model using Keras analyzes the video data and converts the sign language motions into text.

[0958] Step 3:

[0959] The server transmits the converted text data to the smart glasses. The input is the text data from the sign language recognition model, and the output is the text displayed on the display of the smart glasses. Specifically, the server converts the text data into a format for display on the display of the smart glasses and transmits it.

[0960] Step 4:

[0961] The server analyzes the user's facial expressions from the camera image and recognizes emotions. The input is the camera image data, and the output is the recognized emotion data. Specifically, the server uses a facial expression recognition algorithm to analyze the user's facial expressions and detect emotions such as happiness and sadness.

[0962] Step 5:

[0963] The server adjusts the converted text data based on the recognized emotion data. The input is text data and emotion data, and the output is the adjusted text data. Specifically, the server uses a natural language processing library such as TextBlob to modify the text to reflect the emotion.

[0964] Step 6:

[0965] The server retransmits the adjusted text data to the smart glasses, where the input is the adjusted text data and the output is the adjusted text displayed on the display of the smart glasses. Specifically, the server converts the adjusted text data into a format for displaying it again on the display of the smart glasses and transmits it.

[0966] Step 7:

[0967] The user checks the text displayed on the smart glasses display and communicates. The input is the text displayed on the smart glasses display, and the output is the user's understanding and reaction. Specifically, the user reads the displayed text and responds appropriately.

[0968] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0969] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0970] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0971] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0972] [Third embodiment]

[0973] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0974] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0975] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure.

[0976] The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0977] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0978] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0979] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0980] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0981] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0982] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0983] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0984] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0985] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0986] "Example 1"

[0987] The system of the present invention captures sign language motions using cameras, sensors, and other means. The captured motion data is analyzed by AI and converted into another sign language or text. For example, it is possible to capture the motion of "hello" in Japanese sign language and convert it into "hello" in international sign language. It is also possible to convert the captured motion into text and display the text "hello." This allows deaf and hearing people to understand each other's statements even when they are participating in the same conference.

[0988] "Example 2"

[0989] The system of the present invention is operated through a user interface. The user makes a sign language motion toward the camera. The system captures the motion and analyzes it using AI. The analysis results are converted into another sign language or text and displayed on the user interface. For example, if a deaf person makes a motion to say "Hello" in Japanese Sign Language, the system captures the motion, converts it into "Hello" in international sign language, and displays it on the screen. Also, if a hearing person says "Hello," the system converts the utterance into text and displays it on the screen.

[0990] The process flow of each embodiment will be described below.

[0991] "Example 1"

[0992] Step 1: The user performs a sign language motion using a camera, sensor, or other means.

[0993] Step 2: The system captures the motion data.

[0994] Step 3: The captured motion data is analyzed by AI.

[0995] Step 4: The analysis results are converted into another sign language or text.

[0996] Step 5: The conversion result is displayed to the user.

[0997] "Example 2"

[0998] Step 1: The user performs a sign language motion through the system's user interface.

[0999] Step 2: The system captures the motion data.

[1000] Step 3: The captured motion data is analyzed by AI.

[1001] Step 4: The analysis results are converted into another sign language or text.

[1002] Step 5: The conversion result will be displayed on the user interface.

[1003] Example 1

[1004] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1005] Conventional sign language interpretation systems have difficulty accurately capturing sign language motions and converting them into other sign languages ​​or text. In addition, when deaf and hearing people participate in the same conference, it is difficult for them to understand each other's statements in real time. This has hindered smooth communication.

[1006] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1007] In this invention, the server includes a means for capturing sign language motion, a means for preprocessing the captured sign language motion, a means for inputting the preprocessed sign language motion into a generative AI model for analysis, a means for converting the analysis result into another sign language or text, and a means for displaying the converted sign language or text. This makes it possible to accurately capture sign language motion and convert it into another sign language or text. In addition, even if a deaf person and a hearing person are participating in the same conference, they can understand each other's remarks in real time.

[1008] A "means for capturing sign language motion" is a device or method for capturing a user's hand movements using a camera or sensor and collecting sign language motion data.

[1009] The "means for pre-processing the captured sign language motion" refers to a device or method that performs processing such as noise removal, data normalization, and frame interpolation on the captured motion data to improve the quality of the data.

[1010] "Means for inputting preprocessed sign language motion into a generative AI model for analysis" refers to a device or method for inputting preprocessed motion data into an AI model and analyzing sign language motion.

[1011] "Means for converting the analysis results into another sign language or text" refers to a device or method for converting the analysis results of an AI model into another sign language or text.

[1012] A "means for displaying the converted sign language or text" is a device or method that visually displays the converted sign language or text to a user.

[1013] "Japanese Sign Language" is a type of sign language used in Japan, a means of communication using specific hand movements and gestures.

[1014] An "international sign language" is a type of sign language used internationally, which is a common means of communication between sign language users in different countries and regions.

[1015] A "generative AI model" is an artificial intelligence model trained to analyze sign language motion data and convert it into other sign languages ​​or text.

[1016] This invention is a system that uses cameras and sensors to capture sign language motions, collects sign language motion data, analyzes it using AI, and converts it into another sign language or text. A specific embodiment of this system will be described below.

[1017] First, the user activates a camera and a sensor (e.g., Microsoft Kinect or Leap Motion) to capture the motion of the sign language, which then captures the user's hand movements in real time.

[1018] The device then transmits the captured sign language motion data to the server, which then pre-processes the received motion data, including noise removal, data normalization, frame interpolation, etc., to improve the quality of the data.

[1019] The preprocessed data is input by the server into a generative AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model analyzes the sign language motion and obtains the analysis result. The analysis result indicates which sign language the sign language motion corresponds to.

[1020] The server converts the analysis results into another sign language (for example, from Japanese Sign Language to an international sign language) or into text. The converted data is sent to the user's device and displayed. Specifically, the text is displayed on the screen, and the sign language motion is reproduced as animation.

[1021] As a concrete example, consider the case of capturing the motion of "Hello" in Japanese Sign Language. A camera or sensor captures the user's hand movements and sends the data to a server. The server uses an AI model to analyze this motion data and converts it into the international sign language "Hello." At the same time, it also converts it into the text "Hello" and displays it on the user's device.

[1022] Example of a prompt to be input to a generative AI model:

[1023] "Please analyze the motion data of "Hello" in Japanese Sign Language and convert it into "Hello" in International Sign Language. Also, please generate "Hello" as text."

[1024] With this system, even if deaf and hearing people are participating in the same conference, they can understand what each other is saying in real time, which will facilitate smooth communication.

[1025] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1026] Step 1:

[1027] A user activates a camera or sensor. A user activates a camera or sensor (e.g., Microsoft Kinect or Leap Motion) to capture sign motion, which captures the user's hand movements in real time. The input is the user's hand movements, and the output is the captured motion data.

[1028] Step 2:

[1029] The device captures the motion data of sign language. The device captures the user's hand movements in real time through the activated camera or sensor. Specifically, it collects hand position and movement data for each frame. The input is the hand movement data from the camera or sensor, and the output is the captured motion data.

[1030] Step 3:

[1031] The device transmits the captured data to the server. The device transmits the captured motion data to the server. This data includes hand position information, motion vectors, etc. The input is the captured motion data, and the output is the data transmitted to the server.

[1032] Step 4:

[1033] The server preprocesses the motion data. The server preprocesses the motion data it receives. Preprocessing includes noise removal, data normalization, frame interpolation, etc. This improves the quality of the data. The input is the motion data sent to the server, and the output is the preprocessed data.

[1034] Step 5:

[1035] The server inputs the preprocessed data into a generative AI model. The server inputs the preprocessed data into an AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model is trained to analyze sign language motion. The input is the preprocessed motion data, and the output is the analysis result of the AI ​​model.

[1036] Step 6:

[1037] The server obtains the analysis results of the AI ​​model. The server obtains the analysis results from the AI ​​model. The analysis results indicate which sign language the sign language motion corresponds to. The input is the analysis results of the AI ​​model, and the output is the analyzed sign language data.

[1038] Step 7:

[1039] The server converts the analysis results into another sign language or text. Based on the analysis results, the server converts into another sign language (for example, from Japanese Sign Language to international sign language) or text. Specifically, it generates motion data and text data for the corresponding sign language. The input is the analyzed sign language data, and the output is the converted sign language or text data.

[1040] Step 8:

[1041] The server sends the conversion result to the terminal. The server sends the converted sign language or text data to the terminal. The input is the converted sign language or text data, and the output is the data sent to the terminal.

[1042] Step 9:

[1043] The terminal displays the conversion result. The terminal displays the received sign language or text data to the user. Specifically, it displays the text on the screen or reproduces the sign language motion with animation. The input is the sign language or text data sent to the terminal, and the output is the information displayed to the user.

[1044] (Application example 1)

[1045] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1046] There is a problem that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult for hearing-impaired people to communicate with hearing-nonhearing people who cannot understand sign language, which often causes inconvenience when serving customers or giving directions. In addition, there is a lack of technology to convert sign language into voice or text, making real-time communication difficult.

[1047] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1048] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for analyzing the captured sign language motion and converting it into voice, and a means for converting voice into text or sign language and displaying it. This enables hearing-impaired people and hearing-normal people to communicate smoothly in a brick-and-mortar store.

[1049] A "means for capturing sign language motion" is a device or technology that uses a camera or sensor to capture sign language movements.

[1050] "Means for converting captured sign language motion into another sign language or text" refers to technology that uses AI or machine learning models to convert captured sign language motion into another sign language format or text.

[1051] A "means for displaying converted signs or text" is a device or technology that uses a display or screen to visually present the converted signs or text.

[1052] "Means of analyzing captured sign language motion and converting it into audio" refers to technology that uses AI and voice synthesis technology to convert captured sign language movements into audio.

[1053] "Means for converting speech into text or sign language and displaying it" refers to a device or technology that uses speech recognition technology to convert speech into text or sign language and display it visually.

[1054] A system for implementing this invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for analyzing the captured sign language motion and converting it into speech, and means for converting speech into text or sign language and displaying it.

[1055] 1. System Programming

[1056] The program for this system is implemented using Python. Specifically, it uses OpenCV to capture sign language motion from the camera, loads the AI ​​model using Keras, and analyzes the sign language motion and converts it into text and audio.

[1057] 2. Program Processing

[1058] The server captures the sign language motion using a camera (e.g. Logitech C920). The captured video is pre-processed using OpenCV and resized to 64x64 pixels. It is then input to an AI model loaded using Keras to analyze the sign language motion. The analysis results are converted into text or audio and displayed or played through a display or speaker.

[1059] 3. Specific Examples

[1060] For example, consider a customer service scenario in a brick-and-mortar store. When a hearing-impaired person wears smart glasses and says "hello" in sign language, the camera captures their motion, and the AI ​​analyzes it to display the text "hello" on the smart glasses' display. Conversely, when a store clerk says "Welcome," the speech is converted into text and displayed as sign language.

[1061] 4. Examples of prompts

[1062] Here is an example of a prompt to be input to the generative AI model:

[1063] "I want to develop a sign language interpretation application. I want to create a system that captures sign language motion with a camera, analyzes it with AI, and converts it into text. Can you please tell me a program that uses Python, OpenCV, and Keras to capture sign language motion and analyze it with an AI model?"

[1064] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1065] Step 1:

[1066] The server captures the sign language motion using a camera. The input is video data from the camera, and the output is the captured sign language motion video. Specifically, the camera captures video in real time and transmits the video to the server frame by frame.

[1067] Step 2:

[1068] The server preprocesses the captured video using OpenCV. The input is the captured sign language motion video, and the output is the preprocessed video data. Specifically, the video is resized to 64x64 pixels and converted into a format that can be input to the AI ​​model.

[1069] Step 3:

[1070] The server inputs the preprocessed video data into the AI ​​model loaded using Keras. The input is the preprocessed video data, and the output is the sign language motion analysis result. In concrete terms, the AI ​​model analyzes the video data and generates prediction results for converting the sign language motion into text and audio.

[1071] Step 4:

[1072] The server converts the analysis results into text or audio. The input is the prediction result of the AI ​​model, and the output is text data or audio data. Specifically, it generates text based on the prediction result, and generates audio using speech synthesis technology if necessary.

[1073] Step 5:

[1074] The server displays or plays the generated text or audio through a display or speaker. The input is text data or audio data, and the output is visual or auditory information. As a specific operation, the text is displayed on the display and the audio is played from the speaker.

[1075] Step 6:

[1076] The user sees the text shown on the display or the audio played from the speaker. The input is the visual or auditory information, and the output is the user's understanding. In concrete terms, the user reads the displayed text or listens to the played audio.

[1077] Example 2

[1078] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1079] Conventional sign language recognition systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. They also lacked the functionality to convert speech into text and display it in a format that can be read by hearing-impaired people. This has led to problems with smooth communication between hearing-impaired people and hearing-able people.

[1080] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1081] In this invention, the server includes means for capturing sign language motion, means for analyzing the captured sign language motion, means for converting the analyzed sign language motion into another sign language or text, means for displaying the converted sign language or text, means for capturing voice, means for converting the captured voice into text, and means for displaying the converted text. This makes it possible to accurately analyze sign language motion and convert it into another sign language or text, and further convert voice into text and display it, enabling smooth communication between hearing-impaired people and hearing-normal people.

[1082] The "means for capturing sign language motion" refers to a device or method for capturing the sign language movements made by a user as video data using a photographing device such as a camera.

[1083] "Means for analyzing the captured sign language motion" refers to an AI model or algorithm for analyzing the captured video data and identifying the content of the sign language.

[1084] "Means for converting analyzed sign language motion into another sign language or text" refers to a device or method for converting the sign language content into another sign language or text format based on the analysis results.

[1085] A "means for displaying the converted sign language or text" is a display or software for displaying the converted sign language or text in a user interface.

[1086] The "means for capturing voice" refers to a device or method for capturing a user's speech as voice data using a voice input device such as a microphone.

[1087] "Means for converting captured speech into text" refers to speech recognition software or algorithms used to analyze the captured speech data and convert it into text form.

[1088] "Means for displaying the converted text" refers to a display or software for displaying the converted text on a user interface.

[1089] This invention is a system that captures sign language motion, analyzes it, and converts it into another sign language or text. It also includes a function to capture voice, convert it into text, and display it. A specific embodiment of this system will be described below.

[1090] Hardware and Software Used

[1091] Hardware:

[1092] Camera: A high-resolution camera is used to capture the user's sign language motions.

[1093] Microphone: An audio input device for capturing the user's voice.

[1094] Display: A screen for displaying the user interface.

[1095] software:

[1096] AI model for sign language recognition: Analyzes sign language motions using machine learning frameworks such as TensorFlow and PyTorch.

[1097] Speech Recognition Software: Convert speech to text using speech recognition technology such as Google Speech-to-Text.

[1098] System Operation

[1099] Sign Language Motion Capture and Analysis:

[1100] When a user performs a sign language motion in front of the camera, the device captures the motion. The captured video data is sent to a server, which analyzes the video data using an AI model for sign language recognition. The analysis results are converted into another sign language or text and sent to the device. The device displays the conversion results on the user interface.

[1101] Audio capture and conversion:

[1102] When a user speaks, the device captures the speech using a microphone. The captured speech data is sent to a server, which uses voice recognition software to convert the speech into text. The converted text is sent to the device and displayed on a user interface.

[1103] Examples

[1104] Sign language motion capture and conversion:

[1105] The user makes the motion to say "hello" in Japanese sign language in front of the camera.

[1106] The device captures the motion using a camera and transmits the video data to a server.

[1107] The server inputs the video data into an AI model for sign language recognition and analyzes it.

[1108] The server converts the analysis result into the international sign language "hello" and sends it to the terminal.

[1109] The terminal displays the conversion result on the user interface.

[1110] Audio capture and conversion:

[1111] The user says "Hello."

[1112] The terminal captures the audio using a microphone and transmits the audio data to the server.

[1113] The server inputs the voice data into speech recognition software and converts it into text.

[1114] The server transmits the conversion result to the terminal.

[1115] The terminal displays the conversion result on the user interface.

[1116] Examples of prompt statements

[1117] "If I make a motion to say 'hello' in Japanese sign language, please convert it into international sign language."

[1118] "If you say 'hello' by voice, convert that speech to text."

[1119] In this way, the server and the terminal work together to realize a system that analyzes the user's sign language and voice, converts it into an appropriate format, and displays it.

[1120] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1121] Step 1:

[1122] The user makes sign language motions towards the camera.

[1123] As a specific action, a user makes a motion to say "hello" in Japanese sign language in front of the camera. The input is the user's sign language motion, and the output is the video data captured by the camera.

[1124] Step 2:

[1125] The device uses a camera to capture sign language motions.

[1126] Specifically, the device captures the user's sign language motion as video data using a high-resolution camera. The input is the user's sign language motion, and the output is the captured video data.

[1127] Step 3:

[1128] The terminal transmits the captured video data to the server.

[1129] In particular, the terminal transmits video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[1130] Step 4:

[1131] The server inputs the video data into an AI model for sign language recognition.

[1132] Specifically, the server inputs the received video data into an AI model for sign language recognition (e.g., TensorFlow or PyTorch). The input is the video data sent to the server, and the output is the data input into the AI ​​model.

[1133] Step 5:

[1134] The server uses an AI model to analyze sign language motions.

[1135] Specifically, the server uses the AI ​​model to analyze the video data and identify the sign language content. The input is the video data input to the AI ​​model, and the output is the analyzed sign language content.

[1136] Step 6:

[1137] The server converts the analysis results into another sign language or text.

[1138] Specifically, the server converts the sign language content into another sign language (e.g., international sign language) or text format based on the analysis results. The input is the analyzed sign language content, and the output is the converted sign language or text.

[1139] Step 7:

[1140] The server transmits the conversion result to the terminal.

[1141] Specifically, the server sends the conversion result to the terminal. The input is the converted sign language or text, and the output is the conversion result sent to the terminal.

[1142] Step 8:

[1143] The terminal displays the conversion result on the user interface.

[1144] As a specific operation, the terminal displays the conversion result on a user interface. The input is the conversion result sent to the terminal, and the output is the sign language or text displayed on the user interface.

[1145] Step 9:

[1146] The user speaks aloud.

[1147] As a specific operation, the user says "Hello." The input is the user's voice, and the output is the voice data captured by the microphone.

[1148] Step 10:

[1149] The device captures audio using a microphone.

[1150] In particular, the terminal uses a microphone to capture the user's voice as voice data, with the input being the user's voice and the output being the captured voice data.

[1151] Step 11:

[1152] The device transmits the captured audio data to the server.

[1153] In particular, the terminal transmits voice data to the server in real time, where the input is the captured voice data and the output is the voice data transmitted to the server.

[1154] Step 12:

[1155] The server inputs the voice data into voice recognition software.

[1156] Specifically, the server inputs the received voice data into speech recognition software (e.g. Google Speech-to-Text). The input is the voice data sent to the server, and the output is the data input into the speech recognition software.

[1157] Step 13:

[1158] The server converts the voice data into text.

[1159] Specifically, the server uses speech recognition software to convert the voice data into text, with the input being the voice data entered into the speech recognition software and the output being the converted text.

[1160] Step 14:

[1161] The server transmits the conversion result to the terminal.

[1162] In concrete operation, the server sends the conversion result to the terminal, where the input is the converted text and the output is the conversion result sent to the terminal.

[1163] Step 15:

[1164] The terminal displays the conversion result on the user interface.

[1165] In particular, the terminal displays the conversion result on a user interface, where the input is the conversion result sent to the terminal and the output is the text displayed on the user interface.

[1166] (Application example 2)

[1167] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1168] Conventional sign language translation systems have limited functionality for converting sign language into text and translating between different sign languages, making it difficult for deaf and hearing people to communicate smoothly. In addition, there is a lack of means for deaf and hearing people to communicate smoothly in physical stores.

[1169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1170] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for converting voice into text, a means for displaying the converted text, and a means for translating between different sign languages, thereby enabling deaf and hearing people to communicate smoothly in a brick-and-mortar store.

[1171] A "sign language motion capture means" is a device or method that uses a camera or sensor to capture the sign language movements made by a user.

[1172] A "means for converting captured sign language motion into another sign language or text format" is software or algorithms that analyzes captured sign language motions and converts them into another sign language or text format.

[1173] A "means for displaying the converted sign language or text" is a display or screen that visually displays the converted sign language or text in a user interface.

[1174] A "speech-to-text conversion means" is software or algorithms used to convert speech data into text data using speech recognition technology.

[1175] An "interlanguage translation tool" is a translation algorithm or software that converts from one sign language to another.

[1176] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for converting speech to text, means for displaying the converted text, and means for translating between different sign languages.

[1177] System Programming

[1178] The system program performs the following processing.

[1179] 1. Means of capturing sign language motion:

[1180] The system uses cameras and sensors to capture the sign language movements made by the user, specifically, cameras installed in smartphones and smart glasses.

[1181] 2. How to convert captured sign motion into other signs or text:

[1182] It uses machine learning libraries such as TensorFlow to analyze the captured sign language movements and convert them into other sign languages ​​or text formats, using pre-trained generative AI models.

[1183] 3. Means for displaying converted signs and / or text:

[1184] Use the display of a smartphone or smart glasses to visually display the converted sign language or text on a user interface.

[1185] 4. Ways to convert speech to text:

[1186] The SpeechRecognition library is used to convert voice data into text data using speech recognition technology, specifically, using the microphones on smartphones and smart glasses.

[1187] 5. Means of translation between different sign languages:

[1188] Using translation algorithms or software to convert from one sign language to another, including using generative AI models to translate.

[1189] Processing flow

[1190] The server takes in video from the camera and captures the sign language motion. The captured video is analyzed using TensorFlow and converted into another sign language or text. The converted sign language or text is displayed on the display of the smartphone or smart glasses. Using voice recognition technology, the voice data is converted into text data, which is also displayed. Translation between different sign languages ​​is also performed using generative AI models.

[1191] Examples

[1192] For example, if a deaf customer says "Hello" in sign language, the camera in the smart glasses will capture that motion, and the AI ​​will convert it into text and display it on the store clerk's smartphone. If the store clerk says "Welcome," the speech will be converted into text and displayed on the deaf customer's smart glasses.

[1193] Examples of prompt statements

[1194] "When a user says 'Hello' in sign language, capture that motion with a camera, analyze it with an AI model, and convert it into text as 'Hello.' Also, when a store clerk says 'Welcome,' convert that speech into text and display it."

[1195] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1196] Step 1:

[1197] A user performs a sign language motion. The camera in a smartphone or smart glasses captures the motion. The input is the user's sign language motion, and the output is the captured video data. In concrete terms, the camera continuously shoots video and captures the sign language motion in real time.

[1198] Step 2:

[1199] The server receives the captured video data and performs preprocessing. The input is the captured video data, and the output is the preprocessed image data. The specific operations include resizing the video data, removing noise, and extracting the sign language motion features.

[1200] Step 3:

[1201] The server inputs the preprocessed image data into the generative AI model to analyze the sign language motion. The input is the preprocessed image data, and the output is the analysis result, a text representation of the sign language. In concrete terms, the generative AI model analyzes the characteristics of the sign language motion and generates the corresponding text.

[1202] Step 4:

[1203] The server converts the results of the analysis into another sign language or text. The input is a textual representation of the sign language, and the output is the converted sign language or text. In particular, a translation algorithm converts the textual representation of the sign language into a different sign language or text.

[1204] Step 5:

[1205] The server sends the converted sign language or text to the terminal and displays it. The input is the converted sign language or text, and the output is the information displayed on the terminal's display. The specific operation is to visually display the converted sign language or text on the terminal's display.

[1206] Step 6:

[1207] A user speaks using voice. The microphone in a smartphone or smart glasses captures the voice. The input is the user's voice, and the output is the captured voice data. In concrete terms, the microphone continuously records the voice and captures the voice data in real time.

[1208] Step 7:

[1209] The server receives the captured voice data and performs speech recognition. The input is the captured voice data, and the output is text data that is the result of speech recognition. In concrete terms, the speech recognition algorithm analyzes the voice data and generates corresponding text.

[1210] Step 8:

[1211] The server sends the speech recognition results to the terminal and displays them. The input is the text data of the speech recognition results, and the output is the information displayed on the terminal's display. The specific operation is to visually display the speech recognition results on the terminal's display.

[1212] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1213] "Example 1"

[1214] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes the user's emotion from the captured sign motion. Specifically, the emotion engine determines the likelihood that the user is expressing an emotion such as joy, anger, sadness, surprise, etc. from the speed, strength, and rhythm of the sign motion. The emotion engine adjusts the sign-to-text conversion according to the emotion. For example, if it is determined that the user is expressing an emotion of anger, an expression reflecting that emotion can be included in the text.

[1215] "Example 2"

[1216] In another embodiment of the present invention, an emotion engine is incorporated to recognize the emotion of the user from the captured sign motion and adjust the conversion from Japanese Sign Language to International Sign Language based on the emotion. Specifically, if it is determined that the user is expressing an emotion of joy, an expression reflecting the emotion can be included in the international sign language.

[1217] The process flow of each embodiment will be described below.

[1218] "Example 1"

[1219] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[1220] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[1221] Step 3: The emotion engine adjusts the sign-to-text conversion based on the determined emotion. For example, if it determines that the user is expressing an emotion of anger, it can include a wording that reflects that emotion in the text.

[1222] "Example 2"

[1223] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[1224] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[1225] Step 3: The emotion engine adjusts the conversion from Japanese Sign Language to international sign language based on the determined emotion. For example, if it determines that the user is expressing the emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[1226] Example 1

[1227] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1228] Conventional sign language translation systems were able to capture sign language motions and convert them into other sign languages ​​or text, but they lacked the ability to recognize the user's emotions and adjust the translation results accordingly. This made it difficult to accurately convey the nuances and emotions of sign language, resulting in a decline in the quality of communication.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1230] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion from the captured sign language motion, and a means for adjusting the conversion from sign language to text in accordance with the recognized emotion. This enables sign language translation that reflects the user's emotion, realizing more natural and accurate communication.

[1231] A "means for capturing sign language motion" is a device or method that uses a camera or motion sensor to capture a user's sign language movements in real time and acquire them as digital data.

[1232] "Means for converting captured sign language motion into another sign language or text" refers to a device or method that uses an AI analysis engine to analyze acquired sign language motion data and convert it into a corresponding other sign language or text.

[1233] "Means for displaying the converted sign language or text" refers to a device or method, such as a display or monitor, for visually presenting the converted sign language or text to a user.

[1234] The "means for recognizing a user's emotions from the captured sign language motion" refers to an emotion engine or algorithm that analyzes characteristics of the sign language motion, such as speed, strength, and rhythm, and estimates the user's emotions.

[1235] "Means for adjusting the sign language-to-text conversion in response to recognized emotions" refers to an apparatus or method that adjusts the sign language-to-text conversion result based on the user's emotions recognized by an emotion engine, and generates text that reflects the emotions.

[1236] The system of the present invention includes a camera and a motion sensor for capturing the motion of the sign language, an AI analysis engine for analyzing the captured motion data, an emotion engine for recognizing the emotion of the user, and a display for displaying the converted sign language and text. Specific embodiments are described below.

[1237] Hardware and Software Used

[1238] Hardware: High resolution camera, motion sensor, display

[1239] Software: AI analysis engine, emotion engine, sign language conversion algorithm

[1240] Data Import

[1241] When a user signs, the device's camera and motion sensor capture the sign language motion in real time. The camera captures hand movements in high resolution, and the motion sensor records the hand position and movement in detail. The captured motion data is then sent from the device to a server.

[1242] 4. Data Analysis

[1243] The server inputs the received motion data into an AI analysis engine, which uses a deep learning model to analyze the motion data and convert it into the corresponding sign language or text, for example converting "hello" in Japanese sign language to "hello" in international sign language.

[1244] emotion recognition

[1245] The server inputs the analyzed sign language data into the emotion engine. The emotion engine extracts features of the sign language motion, such as speed, strength, and rhythm, to recognize the user's emotion. For example, if the sign language is fast and strong, it determines that the user is expressing the emotion of anger. It adjusts the conversion from sign language to text according to the recognized emotion. For example, if the user is expressing the emotion of joy, it generates the text "Hello!" that reflects that emotion.

[1246] Data Output

[1247] The server sends the converted sign language or text to the terminal, which displays the received data to the user. For example, the text "Hello!" is displayed on the terminal screen.

[1248] Examples

[1249] For example, when a user wants to say "hello" in Japanese Sign Language, the following specific actions are performed:

[1250] 1. The user signs.

[1251] 2. The device's camera and motion sensors capture the user's sign language motions in real time.

[1252] 3. The device transmits the captured motion data to the server.

[1253] 4. The server inputs the received motion data into the AI ​​analysis engine.

[1254] 5. The AI ​​analysis engine analyzes the motion data and converts the Japanese sign language "hello" into the international sign language "hello."

[1255] 6. The server inputs the analyzed sign language data into the emotion engine.

[1256] 7. The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[1257] 8. The emotion engine determines that the user is expressing the emotion of happiness and generates text that reflects that emotion: "Hello!"

[1258] 9. The server sends the converted sign language or text to the terminal.

[1259] 10. The device displays the received data to the user, for example displaying the text "Hello!" on the device screen.

[1260] Examples of prompt statements

[1261] Examples of prompt sentences to be input to a generative AI model include the following:

[1262] "Analyze the sign language motion data captured by the camera and convert the Japanese sign language "Hello" into the international sign language "Hello." Also, recognize the user's emotions, and if the emotion includes joy, display "Hello!" in the text."

[1263] In this way, the system can analyze sign language motions in real time and convert them into appropriate sign language and text.

[1264] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1265] Step 1:

[1266] The user signs.

[1267] As a specific action, the user says "hello" in Japanese Sign Language.

[1268] Input: User's sign language motion

[1269] Output: None

[1270] Step 2:

[1271] The device's camera and motion sensors capture the user's sign language movements in real time.

[1272] Specifically, the camera captures hand movements in high resolution, while the motion sensor records the hand's position and movement in detail.

[1273] Input: User's sign language motion

[1274] Output: Motion data

[1275] Step 3:

[1276] The terminal transmits the captured motion data to the server.

[1277] Specifically, data is transmitted in real time, minimizing latency.

[1278] Input: Motion data

[1279] Output: Motion data sent to the server

[1280] Step 4:

[1281] The server inputs the received motion data into an AI analysis engine.

[1282] Specifically, the AI ​​analytics engine uses deep learning models to analyze motion data.

[1283] Input: Motion data sent to the server

[1284] Output: Analysis results (sign language and text)

[1285] Step 5:

[1286] The server inputs the parsed sign language data into an emotion engine.

[1287] As a specific action, the emotion engine extracts features such as speed, strength and rhythm of sign language motion.

[1288] Input: Analysis results (sign language or text)

[1289] Output: Emotion data

[1290] Step 6:

[1291] The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[1292] Specifically, if the sign language is fast and forceful, it is determined that the user is expressing anger.

[1293] Input: Emotion data

[1294] Output: Recognized emotion

[1295] Step 7:

[1296] The emotion engine adjusts the sign language to text conversion depending on the recognized emotion.

[1297] Specifically, if the user expresses an emotion of joy, the system generates text that reflects that emotion: "Hello!"

[1298] Input: Recognized emotion

[1299] Output: Adjusted text

[1300] Step 8:

[1301] The server sends the converted sign language or text to the terminal.

[1302] Specifically, the data is encrypted and transmitted securely.

[1303] Input: Adjusted text

[1304] Output: Text sent to the terminal.

[1305] Step 9:

[1306] The terminal displays the received data to the user.

[1307] The specific action is that the text "Hello!" will be displayed on the device screen.

[1308] Input: Text sent to the terminal

[1309] Output: The displayed text

[1310] (Application example 1)

[1311] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1312] There is an issue that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult to convey information using sign language to hearing-nonhearing people who cannot understand sign language. In addition, there is a lack of technology that can recognize the user's emotions from the motion of sign language and convert it into text or voice that appropriately reflects those emotions, which makes it difficult to communicate accurately with emotions.

[1313] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1314] In this invention, the server includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotion, and means for reading out the converted text aloud. This enables hearing-impaired people and hearing-normal people to communicate smoothly in physical stores.

[1315] "Sign language motion" refers to the movements of the hands, arms, fingers, etc. when using sign language.

[1316] "Means of capturing" refers to devices or methods that use cameras, sensors, etc. to capture sign language motion as digital data.

[1317] A "conversion method" refers to software or algorithms used to convert captured sign motion into another sign language or text.

[1318] "Display device" means a display or screen for visually presenting the converted sign language or text.

[1319] "Means of recognizing emotions" refers to software or algorithms for analyzing and recognizing a user's emotions from sign language motions.

[1320] "Means for tailoring the conversion" means software or algorithms that appropriately adjust the conversion of sign language to text based on recognized emotions.

[1321] "A reading means" refers to a speaker or speech synthesis software for outputting the converted text as speech.

[1322] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotions from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotions, and means for reading aloud the converted text.

[1323] Hardware and Software Configuration

[1324] Hardware

[1325] Smart glasses: Equipped with cameras and sensors to capture sign language motions.

[1326] Computer: Built into the smart glasses, it processes data.

[1327] software

[1328] OpenCV: For capturing and pre-processing camera images.

[1329] Keras: Runs AI models for sign language and emotion recognition.

[1330] pyttsx3: Used as a text-to-speech engine.

[1331] Data processing and calculation

[1332] Sign language motion capture

[1333] We use the camera and sensors installed in the smart glasses to capture the user's sign language motion in real time. We use OpenCV to acquire and pre-process the video.

[1334] Sign language motion conversion

[1335] The preprocessed video data is fed into a Keras-based sign language recognition model to convert the sign language motion into text. It is also possible to convert it into another sign language (e.g., from Japanese Sign Language to International Sign Language).

[1336] Emotion recognition

[1337] The same video data is then fed into an emotion recognition model using Keras to analyze the user's emotions. The model recognizes emotions from the speed, strength, and rhythm of the sign language motion.

[1338] Text Adjustment

[1339] Adjust the sign-to-text conversion based on the recognized emotion. For example, if the user is expressing anger, include a statement in the text that reflects that emotion.

[1340] View and read text aloud

[1341] The converted text is displayed on the smart glasses display and then read aloud using pyttsx3.

[1342] Examples

[1343] When a hearing-impaired person uses sign language to ask a store clerk, "Where is this item?", the smart glasses will recognize the sign language and tell the store clerk, "Where is this item?" Furthermore, the emotion engine will recognize that the user is in a hurry based on the speed and strength of the sign language, and add, "You seem to be in a hurry."

[1344] Examples of prompt statements

[1345] The customer signs, "Where is this item?". From the speed and force of the sign, you recognize that the customer is in a hurry, so you verbally tell the store associate, "Where is this item? It seems like you're in a hurry."

[1346] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1347] Step 1:

[1348] A camera mounted on the smart glasses captures the user's sign language motion in real time. The input is the camera image, and the output is the captured image data. The camera image is acquired using OpenCV.

[1349] Step 2:

[1350] The server pre-processes the captured video data. The input is the captured video data and the output is the pre-processed video data. Pre-processing includes video resizing and normalization.

[1351] Step 3:

[1352] The server inputs the preprocessed video data into a sign language recognition model and converts the sign language motions into text. The input is the preprocessed video data and the output is the converted text. The sign language recognition model is implemented using Keras.

[1353] Step 4:

[1354] The server inputs the same preprocessed video data into an emotion recognition model to analyze the user's emotions. The input is the preprocessed video data, and the output is the recognized emotion data. The emotion recognition model is also implemented using Keras.

[1355] Step 5:

[1356] The server adjusts the sign-to-text conversion based on the recognized emotion data. The input is the converted text and the recognized emotion data, and the output is the adjusted text that reflects the emotion. The adjustment includes changing the expression of the text according to the emotion.

[1357] Step 6:

[1358] The server displays the adjusted text on the display of the smart glasses. The input is the adjusted text and the output is the text displayed on the display.

[1359] Step 7:

[1360] The server reads the adjusted text aloud. The input is the adjusted text, and the output is the audio data. The server uses pyttsx3 for the voice reading.

[1361] Example 2

[1362] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1363] Conventional sign language translation systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. In addition, the translation does not take into account the user's emotions, which leads to a problem of poor communication quality. Furthermore, there is a lack of means for able-bodied people to understand sign language.

[1364] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1365] In this invention, the server includes a means for capturing sign language motion, a means for analyzing the captured sign language motion, a means for converting the analysis result into another sign language or text, a means for displaying the converted sign language or text, and a means for recognizing the user's emotion and adjusting the conversion result based on the emotion. This makes it possible to accurately analyze the sign language motion and perform translation that takes the user's emotion into account, and also provides a means for able-bodied people to understand sign language.

[1366] "Sign language motion" refers to the movement and position of the hands and fingers when signing.

[1367] "Means of capturing" refers to devices or methods for capturing sign language motion as digital data using cameras, sensors, etc.

[1368] "Means of analysis" refers to devices or methods for analyzing captured sign language motions using AI or machine learning models and understanding their meaning.

[1369] "Means for converting" refers to a device or method that converts sign language motion into another sign language or text based on the analysis results.

[1370] "Means for displaying" refers to a device or method for displaying the converted sign language or text on a user interface.

[1371] "Means for recognizing emotions" refers to a device or method for analyzing emotions from a user's sign language motions and facial expressions and understanding those emotions.

[1372] "Means for adjusting" refers to a device or method that appropriately modifies the conversion result based on the recognized emotion.

[1373] "Japanese Sign Language" refers to a type of sign language used in Japan.

[1374] "International sign language" refers to a type of sign language that is internationally accepted.

[1375] "Text" refers to character information.

[1376] "Normal person" refers to a person without hearing impairment.

[1377] The present invention is a system for capturing and analyzing sign language motion, converting it into another sign language or text, and displaying it. A specific embodiment of this system will be described below.

[1378] First, the user makes a sign language motion toward the camera. The device uses a camera (e.g., a general web camera) to capture the user's sign language motion as video data. This video data is then sent to the server in real time.

[1379] The server sends the received video data to an AI analysis engine (for example, a custom model using TensorFlow). The AI ​​analysis engine analyzes the sign language motion and understands its meaning. For example, it analyzes the motion of "hello" in Japanese sign language and obtains the result.

[1380] Next, the server converts the sign language motion into another sign language (e.g., international sign language) or text based on the analysis results. The converted sign language or text is displayed on a user interface (e.g., a web application built with React.js). The user can check the conversion results on the screen.

[1381] In addition, the system has built-in emotion recognition capabilities. The server uses an emotion recognition engine (e.g., Microsoft Azure Emotion API) to recognize emotions from the captured sign motion and the user's facial expressions. Based on the recognized emotion, the conversion result can be adjusted appropriately. For example, if the user is expressing the emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[1382] As a concrete example, consider the case where a deaf person makes a motion to say "Hello" in Japanese Sign Language. The device captures that motion with a camera and sends the video data to the server. The server sends the video data to an AI analysis engine and obtains the analysis results. The server then converts the analysis results into "Hello" in international sign language and sends it to the user interface. The device displays the conversion result on the screen. The user can see "Hello" in international sign language on the screen.

[1383] Also, when a healthy person says "Hello," the device captures the voice data with a microphone (e.g., a general USB microphone) and sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). The voice recognition engine converts the utterance into text and displays it on the user interface. The user can see the text "Hello" on the screen.

[1384] An example of a prompt sentence might be:

[1385] "Analyze the sign language motions made by the user facing the camera and convert them from Japanese Sign Language to international sign language. Also, recognize the user's emotions and adjust the conversion results based on those emotions."

[1386] Using this prompt, the generative AI model can analyze the user's sign language motions and emotions and provide appropriate conversion results.

[1387] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1388] Step 1:

[1389] The user makes sign language motions towards the camera.

[1390] As a specific action, the user performs a motion to say "hello" in Japanese sign language. The input is the user's sign language motion, and the output is the video data captured by the camera.

[1391] Step 2:

[1392] The device uses a camera to capture sign language motions.

[1393] In particular, the terminal uses a camera (e.g., a general webcam) to capture the user's sign language motion as video data. The input is the user's sign language motion, and the output is the captured video data.

[1394] Step 3:

[1395] The terminal transmits the captured video data to the server.

[1396] In particular, the terminal transmits captured video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[1397] Step 4:

[1398] The server sends the video data to an AI analysis engine.

[1399] Specifically, the server sends the received video data to an AI analysis engine (e.g., a custom model using TensorFlow). The input is the video data sent to the server, and the output is the video data sent to the AI ​​analysis engine.

[1400] Step 5:

[1401] An AI analysis engine analyzes sign language motions.

[1402] Specifically, the AI ​​analysis engine analyzes video data and recognizes sign language motions. The input is the video data sent to the AI ​​analysis engine, and the output is the analysis results. For example, the motion of "hello" is recognized as Japanese sign language.

[1403] Step 6:

[1404] The server converts the analysis results into another sign language or text.

[1405] Specifically, the server receives the analysis results from the AI ​​analysis engine and converts them into another sign language (for example, international sign language) or text. The input is the analysis result, and the output is the converted sign language or text. For example, "Hello" in Japanese Sign Language is converted into "Hello" in international sign language.

[1406] Step 7:

[1407] The server sends the conversion result to the user interface.

[1408] In particular, the server sends the conversion result to the user interface. The input is the converted sign language or text, and the output is the conversion result sent to the user interface.

[1409] Step 8:

[1410] The terminal displays the conversion result on the user interface.

[1411] Specifically, the terminal displays the conversion result on a user interface (e.g., a web application built with React.js). The input is the conversion result sent to the user interface, and the output is the conversion result displayed on the screen. The user can check the conversion result on the screen.

[1412] (Application example 2)

[1413] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1414] Conventional sign language translation systems can capture sign language motions and convert them into other sign languages ​​or text, but they lack the ability to recognize the user's emotions and adjust the conversion based on those emotions, which reduces the quality of communication. In addition, there is a lack of appropriate tools to enable smooth communication between hearing-impaired and hearing-speaking people in brick-and-mortar stores.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1416] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion, a means for adjusting the conversion based on the recognized emotion, and a means for being installed in the smart glasses. This enables conversion of sign language or text that reflects the user's emotion, realizing smooth communication between hearing-impaired and hearing-normal people in brick-and-mortar stores.

[1417] "Sign language motion" refers to the movement and position of the hands and fingers when signing.

[1418] "Means of capturing" refers to devices or methods for capturing sign language motion using cameras, sensors, etc.

[1419] "Means for converting" refers to software or algorithms used to convert captured sign language motion into another sign language or text.

[1420] "Display means" refers to a display or screen that visually presents the converted sign language or text to the user.

[1421] "Means of recognizing user emotions" refers to software or algorithms for analyzing emotions from the user's facial expressions and movements.

[1422] "Means for adjusting the conversion based on recognized emotions" refers to software or algorithms that appropriately modify the sign language or text conversion results by taking into account the user's emotions.

[1423] "Means installed on the smart glasses" refers to a method of incorporating an application or software into the smart glasses.

[1424] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion, means for adjusting the conversion based on the recognized emotion, and means for being installed on smart glasses.

[1425] The server uses a camera to capture the sign language motion. The camera captures the sign language motion of the user in real time and sends the video data to the server. The server preprocesses the video data using an image processing library such as OpenCV and inputs it into the sign language recognition model.

[1426] The sign language recognition model is built using deep learning frameworks such as Keras to analyze sign motion and convert it into corresponding text or another sign language, which is then displayed on the smart glasses' display.

[1427] In addition, the server uses facial expression recognition algorithms to recognize the user's emotions, for example by analyzing the user's facial expressions to detect emotions such as happiness or sadness. The emotion engine uses natural language processing libraries such as TextBlob to adjust the conversion results based on the detected emotions.

[1428] The smart glasses are equipped with a display to visually present the converted sign language and text to the user. When the user signs "Hello," the motion is captured by the camera and analyzed by the sign language recognition model. The analysis result is displayed as "Hello" on the smart glasses display. Furthermore, if the user expresses the emotion of joy, the emotion engine reflects that emotion and displays "Hello! Nice to see you!".

[1429] Examples of prompts for a generative AI model include:

[1430] "If the user signs to say 'hello,' analyze that motion and convert it into appropriate text. Or, if the user's emotion is joy, adjust the text to reflect that emotion."

[1431] In this way, a system is provided that enables smooth communication between hearing-impaired and hearing-normal people in brick-and-mortar stores.

[1432] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1433] Step 1:

[1434] The server captures the user's sign language motion in real time using a camera. The input is the camera video data, and the output is the preprocessed video data. Specifically, the server uses OpenCV to split the video data into frames and detect the positions of the hands and fingers.

[1435] Step 2:

[1436] The server inputs the preprocessed video data into a sign language recognition model. The input is the preprocessed video data, and the output is text data corresponding to the sign language motions. Specifically, a deep learning model using Keras analyzes the video data and converts the sign language motions into text.

[1437] Step 3:

[1438] The server transmits the converted text data to the smart glasses. The input is the text data from the sign language recognition model, and the output is the text displayed on the display of the smart glasses. Specifically, the server converts the text data into a format for display on the display of the smart glasses and transmits it.

[1439] Step 4:

[1440] The server analyzes the user's facial expressions from the camera image and recognizes emotions. The input is the camera image data, and the output is the recognized emotion data. Specifically, the server uses a facial expression recognition algorithm to analyze the user's facial expressions and detect emotions such as happiness and sadness.

[1441] Step 5:

[1442] The server adjusts the converted text data based on the recognized emotion data. The input is text data and emotion data, and the output is the adjusted text data. Specifically, the server uses a natural language processing library such as TextBlob to modify the text to reflect the emotion.

[1443] Step 6:

[1444] The server retransmits the adjusted text data to the smart glasses, where the input is the adjusted text data and the output is the adjusted text displayed on the display of the smart glasses. Specifically, the server converts the adjusted text data into a format for displaying it again on the display of the smart glasses and transmits it.

[1445] Step 7:

[1446] The user checks the text displayed on the smart glasses display and communicates. The input is the text displayed on the smart glasses display, and the output is the user's understanding and reaction. Specifically, the user reads the displayed text and responds appropriately.

[1447] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1448] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1449] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1450] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1451] [Fourth embodiment]

[1452] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1453] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1454] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1455] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1456] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1457] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1458] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1459] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1460] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1461] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1462] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1463] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1464] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1465] "Example 1"

[1466] The system of the present invention captures sign language motions using cameras, sensors, and other means. The captured motion data is analyzed by AI and converted into another sign language or text. For example, it is possible to capture the motion of "hello" in Japanese sign language and convert it into "hello" in international sign language. It is also possible to convert the captured motion into text and display the text "hello." This allows deaf and hearing people to understand each other's statements even when they are participating in the same conference.

[1467] "Example 2"

[1468] The system of the present invention is operated through a user interface. The user makes a sign language motion toward the camera. The system captures the motion and analyzes it using AI. The analysis results are converted into another sign language or text and displayed on the user interface. For example, if a deaf person makes a motion to say "Hello" in Japanese Sign Language, the system captures the motion, converts it into "Hello" in international sign language, and displays it on the screen. Also, if a hearing person says "Hello," the system converts the utterance into text and displays it on the screen.

[1469] The process flow of each embodiment will be described below.

[1470] "Example 1"

[1471] Step 1: The user performs a sign language motion using a camera, sensor, or other means.

[1472] Step 2: The system captures the motion data.

[1473] Step 3: The captured motion data is analyzed by AI.

[1474] Step 4: The analysis results are converted into another sign language or text.

[1475] Step 5: The conversion result is displayed to the user.

[1476] "Example 2"

[1477] Step 1: The user performs a sign language motion through the system's user interface.

[1478] Step 2: The system captures the motion data.

[1479] Step 3: The captured motion data is analyzed by AI.

[1480] Step 4: The analysis results are converted into another sign language or text.

[1481] Step 5: The conversion result will be displayed on the user interface.

[1482] Example 1

[1483] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1484] Conventional sign language interpretation systems have difficulty accurately capturing sign language motions and converting them into other sign languages ​​or text. In addition, when deaf and hearing people participate in the same conference, it is difficult for them to understand each other's statements in real time. This has hindered smooth communication.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1486] In this invention, the server includes a means for capturing sign language motion, a means for preprocessing the captured sign language motion, a means for inputting the preprocessed sign language motion into a generative AI model for analysis, a means for converting the analysis result into another sign language or text, and a means for displaying the converted sign language or text. This makes it possible to accurately capture sign language motion and convert it into another sign language or text. In addition, even if a deaf person and a hearing person are participating in the same conference, they can understand each other's remarks in real time.

[1487] A "means for capturing sign language motion" is a device or method for capturing a user's hand movements using a camera or sensor and collecting sign language motion data.

[1488] The "means for pre-processing the captured sign language motion" refers to a device or method that performs processing such as noise removal, data normalization, and frame interpolation on the captured motion data to improve the quality of the data.

[1489] "Means for inputting preprocessed sign language motion into a generative AI model for analysis" refers to a device or method for inputting preprocessed motion data into an AI model and analyzing sign language motion.

[1490] "Means for converting the analysis results into another sign language or text" refers to a device or method for converting the analysis results of an AI model into another sign language or text.

[1491] A "means for displaying the converted sign language or text" is a device or method that visually displays the converted sign language or text to a user.

[1492] "Japanese Sign Language" is a type of sign language used in Japan, a means of communication using specific hand movements and gestures.

[1493] An "international sign language" is a type of sign language used internationally, which is a common means of communication between sign language users in different countries and regions.

[1494] A "generative AI model" is an artificial intelligence model trained to analyze sign language motion data and convert it into other sign languages ​​or text.

[1495] This invention is a system that uses cameras and sensors to capture sign language motions, collects sign language motion data, analyzes it using AI, and converts it into another sign language or text. A specific embodiment of this system will be described below.

[1496] First, the user activates a camera and a sensor (e.g., Microsoft Kinect or Leap Motion) to capture the motion of the sign language, which then captures the user's hand movements in real time.

[1497] The device then transmits the captured sign language motion data to the server, which then pre-processes the received motion data, including noise removal, data normalization, frame interpolation, etc., to improve the quality of the data.

[1498] The preprocessed data is input by the server into a generative AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model analyzes the sign language motion and obtains the analysis result. The analysis result indicates which sign language the sign language motion corresponds to.

[1499] The server converts the analysis results into another sign language (for example, from Japanese Sign Language to an international sign language) or into text. The converted data is sent to the user's device and displayed. Specifically, the text is displayed on the screen, and the sign language motion is reproduced as animation.

[1500] As a concrete example, consider the case of capturing the motion of "Hello" in Japanese Sign Language. A camera or sensor captures the user's hand movements and sends the data to a server. The server uses an AI model to analyze this motion data and converts it into the international sign language "Hello." At the same time, it also converts it into the text "Hello" and displays it on the user's device.

[1501] Example of a prompt to be input to a generative AI model:

[1502] "Please analyze the motion data of "Hello" in Japanese Sign Language and convert it into "Hello" in International Sign Language. Also, please generate "Hello" as text."

[1503] With this system, even if deaf and hearing people are participating in the same conference, they can understand what each other is saying in real time, which will facilitate smooth communication.

[1504] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1505] Step 1:

[1506] A user activates a camera or sensor. A user activates a camera or sensor (e.g., Microsoft Kinect or Leap Motion) to capture sign motion, which captures the user's hand movements in real time. The input is the user's hand movements, and the output is the captured motion data.

[1507] Step 2:

[1508] The device captures the motion data of sign language. The device captures the user's hand movements in real time through the activated camera or sensor. Specifically, it collects hand position and movement data for each frame. The input is the hand movement data from the camera or sensor, and the output is the captured motion data.

[1509] Step 3:

[1510] The device transmits the captured data to the server. The device transmits the captured motion data to the server. This data includes hand position information, motion vectors, etc. The input is the captured motion data, and the output is the data transmitted to the server.

[1511] Step 4:

[1512] The server preprocesses the motion data. The server preprocesses the motion data it receives. Preprocessing includes noise removal, data normalization, frame interpolation, etc. This improves the quality of the data. The input is the motion data sent to the server, and the output is the preprocessed data.

[1513] Step 5:

[1514] The server inputs the preprocessed data into a generative AI model. The server inputs the preprocessed data into an AI model (e.g., a model built with TensorFlow or PyTorch). The AI ​​model is trained to analyze sign language motion. The input is the preprocessed motion data, and the output is the analysis result of the AI ​​model.

[1515] Step 6:

[1516] The server obtains the analysis results of the AI ​​model. The server obtains the analysis results from the AI ​​model. The analysis results indicate which sign language the sign language motion corresponds to. The input is the analysis results of the AI ​​model, and the output is the analyzed sign language data.

[1517] Step 7:

[1518] The server converts the analysis results into another sign language or text. Based on the analysis results, the server converts into another sign language (for example, from Japanese Sign Language to international sign language) or text. Specifically, it generates motion data and text data for the corresponding sign language. The input is the analyzed sign language data, and the output is the converted sign language or text data.

[1519] Step 8:

[1520] The server sends the conversion result to the terminal. The server sends the converted sign language or text data to the terminal. The input is the converted sign language or text data, and the output is the data sent to the terminal.

[1521] Step 9:

[1522] The terminal displays the conversion result. The terminal displays the received sign language or text data to the user. Specifically, it displays the text on the screen or reproduces the sign language motion with animation. The input is the sign language or text data sent to the terminal, and the output is the information displayed to the user.

[1523] (Application example 1)

[1524] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1525] There is a problem that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult for hearing-impaired people to communicate with hearing-nonhearing people who cannot understand sign language, which often causes inconvenience when serving customers or giving directions. In addition, there is a lack of technology to convert sign language into voice or text, making real-time communication difficult.

[1526] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1527] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for analyzing the captured sign language motion and converting it into voice, and a means for converting voice into text or sign language and displaying it. This enables hearing-impaired people and hearing-normal people to communicate smoothly in a brick-and-mortar store.

[1528] A "means for capturing sign language motion" is a device or technology that uses a camera or sensor to capture sign language movements.

[1529] "Means for converting captured sign language motion into another sign language or text" refers to technology that uses AI or machine learning models to convert captured sign language motion into another sign language format or text.

[1530] A "means for displaying converted signs or text" is a device or technology that uses a display or screen to visually present the converted signs or text.

[1531] "Means of analyzing captured sign language motion and converting it into audio" refers to technology that uses AI and voice synthesis technology to convert captured sign language movements into audio.

[1532] "Means for converting speech into text or sign language and displaying it" refers to a device or technology that uses speech recognition technology to convert speech into text or sign language and display it visually.

[1533] A system for implementing this invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for analyzing the captured sign language motion and converting it into speech, and means for converting speech into text or sign language and displaying it.

[1534] 1. System Programming

[1535] The program for this system is implemented using Python. Specifically, it uses OpenCV to capture sign language motion from the camera, loads the AI ​​model using Keras, and analyzes the sign language motion and converts it into text and audio.

[1536] 2. Program Processing

[1537] The server captures the sign language motion using a camera (e.g. Logitech C920). The captured video is pre-processed using OpenCV and resized to 64x64 pixels. It is then input to an AI model loaded using Keras to analyze the sign language motion. The analysis results are converted into text or audio and displayed or played through a display or speaker.

[1538] 3. Specific Examples

[1539] For example, consider a customer service scenario in a brick-and-mortar store. When a hearing-impaired person wears smart glasses and says "hello" in sign language, the camera captures their motion, and the AI ​​analyzes it to display the text "hello" on the smart glasses' display. Conversely, when a store clerk says "Welcome," the speech is converted into text and displayed as sign language.

[1540] 4. Examples of prompts

[1541] Here is an example of a prompt to be input to the generative AI model:

[1542] "I want to develop a sign language interpretation application. I want to create a system that captures sign language motion with a camera, analyzes it with AI, and converts it into text. Can you please tell me a program that uses Python, OpenCV, and Keras to capture sign language motion and analyze it with an AI model?"

[1543] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1544] Step 1:

[1545] The server captures the sign language motion using a camera. The input is video data from the camera, and the output is the captured sign language motion video. Specifically, the camera captures video in real time and transmits the video to the server frame by frame.

[1546] Step 2:

[1547] The server preprocesses the captured video using OpenCV. The input is the captured sign language motion video, and the output is the preprocessed video data. Specifically, the video is resized to 64x64 pixels and converted into a format that can be input to the AI ​​model.

[1548] Step 3:

[1549] The server inputs the preprocessed video data into the AI ​​model loaded using Keras. The input is the preprocessed video data, and the output is the sign language motion analysis result. In concrete terms, the AI ​​model analyzes the video data and generates prediction results for converting the sign language motion into text and audio.

[1550] Step 4:

[1551] The server converts the analysis results into text or audio. The input is the prediction result of the AI ​​model, and the output is text data or audio data. Specifically, it generates text based on the prediction result, and generates audio using speech synthesis technology if necessary.

[1552] Step 5:

[1553] The server displays or plays the generated text or audio through a display or speaker. The input is text data or audio data, and the output is visual or auditory information. As a specific operation, the text is displayed on the display and the audio is played from the speaker.

[1554] Step 6:

[1555] The user sees the text shown on the display or the audio played from the speaker. The input is the visual or auditory information, and the output is the user's understanding. In concrete terms, the user reads the displayed text or listens to the played audio.

[1556] Example 2

[1557] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1558] Conventional sign language recognition systems have had difficulty accurately analyzing sign language motions and converting them into other sign languages ​​or text. They also lacked the functionality to convert speech into text and display it in a format that can be read by hearing-impaired people. This has led to problems with smooth communication between hearing-impaired people and hearing-able people.

[1559] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1560] In this invention, the server includes means for capturing sign language motion, means for analyzing the captured sign language motion, means for converting the analyzed sign language motion into another sign language or text, means for displaying the converted sign language or text, means for capturing voice, means for converting the captured voice into text, and means for displaying the converted text. This makes it possible to accurately analyze sign language motion and convert it into another sign language or text, and further convert voice into text and display it, enabling smooth communication between hearing-impaired people and hearing-normal people.

[1561] The "means for capturing sign language motion" refers to a device or method for capturing the sign language movements made by a user as video data using a photographing device such as a camera.

[1562] "Means for analyzing the captured sign language motion" refers to an AI model or algorithm for analyzing the captured video data and identifying the content of the sign language.

[1563] "Means for converting analyzed sign language motion into another sign language or text" refers to a device or method for converting the sign language content into another sign language or text format based on the analysis results.

[1564] A "means for displaying the converted sign language or text" is a display or software for displaying the converted sign language or text in a user interface.

[1565] The "means for capturing voice" refers to a device or method for capturing a user's speech as voice data using a voice input device such as a microphone.

[1566] "Means for converting captured speech into text" refers to speech recognition software or algorithms used to analyze the captured speech data and convert it into text form.

[1567] "Means for displaying the converted text" refers to a display or software for displaying the converted text on a user interface.

[1568] This invention is a system that captures sign language motion, analyzes it, and converts it into another sign language or text. It also includes a function to capture voice, convert it into text, and display it. A specific embodiment of this system will be described below.

[1569] Hardware and Software Used

[1570] Hardware:

[1571] Camera: A high-resolution camera is used to capture the user's sign language motions.

[1572] Microphone: An audio input device for capturing the user's voice.

[1573] Display: A screen for displaying the user interface.

[1574] software:

[1575] AI model for sign language recognition: Analyzes sign language motions using machine learning frameworks such as TensorFlow and PyTorch.

[1576] Speech Recognition Software: Convert speech to text using speech recognition technology such as Google Speech-to-Text.

[1577] System Operation

[1578] Sign Language Motion Capture and Analysis:

[1579] When a user performs a sign language motion in front of the camera, the device captures the motion. The captured video data is sent to a server, which analyzes the video data using an AI model for sign language recognition. The analysis results are converted into another sign language or text and sent to the device. The device displays the conversion results on the user interface.

[1580] Audio capture and conversion:

[1581] When a user speaks, the device captures the speech using a microphone. The captured speech data is sent to a server, which uses voice recognition software to convert the speech into text. The converted text is sent to the device and displayed on a user interface.

[1582] Examples

[1583] Sign language motion capture and conversion:

[1584] The user makes the motion to say "hello" in Japanese sign language in front of the camera.

[1585] The device captures the motion using a camera and transmits the video data to a server.

[1586] The server inputs the video data into an AI model for sign language recognition and analyzes it.

[1587] The server converts the analysis result into the international sign language "hello" and sends it to the terminal.

[1588] The terminal displays the conversion result on the user interface.

[1589] Audio capture and conversion:

[1590] The user says "Hello."

[1591] The terminal captures the audio using a microphone and transmits the audio data to the server.

[1592] The server inputs the voice data into speech recognition software and converts it into text.

[1593] The server transmits the conversion result to the terminal.

[1594] The terminal displays the conversion result on the user interface.

[1595] Examples of prompt statements

[1596] "If I make a motion to say 'hello' in Japanese sign language, please convert it into international sign language."

[1597] "If you say 'hello' by voice, convert that speech to text."

[1598] In this way, the server and the terminal work together to realize a system that analyzes the user's sign language and voice, converts it into an appropriate format, and displays it.

[1599] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1600] Step 1:

[1601] The user makes sign language motions towards the camera.

[1602] As a specific action, a user makes a motion to say "hello" in Japanese sign language in front of the camera. The input is the user's sign language motion, and the output is the video data captured by the camera.

[1603] Step 2:

[1604] The device uses a camera to capture sign language motions.

[1605] Specifically, the device captures the user's sign language motion as video data using a high-resolution camera. The input is the user's sign language motion, and the output is the captured video data.

[1606] Step 3:

[1607] The terminal transmits the captured video data to the server.

[1608] In particular, the terminal transmits video data to the server in real time, where the input is the captured video data and the output is the video data transmitted to the server.

[1609] Step 4:

[1610] The server inputs the video data into an AI model for sign language recognition.

[1611] Specifically, the server inputs the received video data into an AI model for sign language recognition (e.g., TensorFlow or PyTorch). The input is the video data sent to the server, and the output is the data input into the AI ​​model.

[1612] Step 5:

[1613] The server uses an AI model to analyze sign language motions.

[1614] Specifically, the server uses the AI ​​model to analyze the video data and identify the sign language content. The input is the video data input to the AI ​​model, and the output is the analyzed sign language content.

[1615] Step 6:

[1616] The server converts the analysis results into another sign language or text.

[1617] Specifically, the server converts the sign language content into another sign language (e.g., international sign language) or text format based on the analysis results. The input is the analyzed sign language content, and the output is the converted sign language or text.

[1618] Step 7:

[1619] The server transmits the conversion result to the terminal.

[1620] Specifically, the server sends the conversion result to the terminal. The input is the converted sign language or text, and the output is the conversion result sent to the terminal.

[1621] Step 8:

[1622] The terminal displays the conversion result on the user interface.

[1623] As a specific operation, the terminal displays the conversion result on a user interface. The input is the conversion result sent to the terminal, and the output is the sign language or text displayed on the user interface.

[1624] Step 9:

[1625] The user speaks aloud.

[1626] As a specific operation, the user says "Hello." The input is the user's voice, and the output is the voice data captured by the microphone.

[1627] Step 10:

[1628] The device captures audio using a microphone.

[1629] In particular, the terminal uses a microphone to capture the user's voice as voice data, with the input being the user's voice and the output being the captured voice data.

[1630] Step 11:

[1631] The device transmits the captured audio data to the server.

[1632] In particular, the terminal transmits voice data to the server in real time, where the input is the captured voice data and the output is the voice data transmitted to the server.

[1633] Step 12:

[1634] The server inputs the voice data into voice recognition software.

[1635] Specifically, the server inputs the received voice data into speech recognition software (e.g. Google Speech-to-Text). The input is the voice data sent to the server, and the output is the data input into the speech recognition software.

[1636] Step 13:

[1637] The server converts the voice data into text.

[1638] Specifically, the server uses speech recognition software to convert the voice data into text, with the input being the voice data entered into the speech recognition software and the output being the converted text.

[1639] Step 14:

[1640] The server transmits the conversion result to the terminal.

[1641] In concrete operation, the server sends the conversion result to the terminal, where the input is the converted text and the output is the conversion result sent to the terminal.

[1642] Step 15:

[1643] The terminal displays the conversion result on the user interface.

[1644] In particular, the terminal displays the conversion result on a user interface, where the input is the conversion result sent to the terminal and the output is the text displayed on the user interface.

[1645] (Application example 2)

[1646] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1647] Conventional sign language translation systems have limited functionality for converting sign language into text and translating between different sign languages, making it difficult for deaf and hearing people to communicate smoothly. In addition, there is a lack of means for deaf and hearing people to communicate smoothly in physical stores.

[1648] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1649] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for converting voice into text, a means for displaying the converted text, and a means for translating between different sign languages, thereby enabling deaf and hearing people to communicate smoothly in a brick-and-mortar store.

[1650] A "sign language motion capture means" is a device or method that uses a camera or sensor to capture the sign language movements made by a user.

[1651] A "means for converting captured sign language motion into another sign language or text format" is software or algorithms that analyzes captured sign language motions and converts them into another sign language or text format.

[1652] A "means for displaying the converted sign language or text" is a display or screen that visually displays the converted sign language or text in a user interface.

[1653] A "speech-to-text conversion means" is software or algorithms used to convert speech data into text data using speech recognition technology.

[1654] An "interlanguage translation tool" is a translation algorithm or software that converts from one sign language to another.

[1655] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for converting speech to text, means for displaying the converted text, and means for translating between different sign languages.

[1656] System Programming

[1657] The system program performs the following processing.

[1658] 1. Means of capturing sign language motion:

[1659] The system uses cameras and sensors to capture the sign language movements made by the user, specifically, cameras installed in smartphones and smart glasses.

[1660] 2. How to convert captured sign motion into other signs or text:

[1661] It uses machine learning libraries such as TensorFlow to analyze the captured sign language movements and convert them into other sign languages ​​or text formats, using pre-trained generative AI models.

[1662] 3. Means for displaying converted signs and / or text:

[1663] Use the display of a smartphone or smart glasses to visually display the converted sign language or text on a user interface.

[1664] 4. Ways to convert speech to text:

[1665] The SpeechRecognition library is used to convert voice data into text data using speech recognition technology, specifically, using the microphones on smartphones and smart glasses.

[1666] 5. Means of translation between different sign languages:

[1667] Using translation algorithms or software to convert from one sign language to another, including using generative AI models to translate.

[1668] Processing flow

[1669] The server takes in video from the camera and captures the sign language motion. The captured video is analyzed using TensorFlow and converted into another sign language or text. The converted sign language or text is displayed on the display of the smartphone or smart glasses. Using voice recognition technology, the voice data is converted into text data, which is also displayed. Translation between different sign languages ​​is also performed using generative AI models.

[1670] Examples

[1671] For example, if a deaf customer says "Hello" in sign language, the camera in the smart glasses will capture that motion, and the AI ​​will convert it into text and display it on the store clerk's smartphone. If the store clerk says "Welcome," the speech will be converted into text and displayed on the deaf customer's smart glasses.

[1672] Examples of prompt statements

[1673] "When a user says 'Hello' in sign language, capture that motion with a camera, analyze it with an AI model, and convert it into text as 'Hello.' Also, when a store clerk says 'Welcome,' convert that speech into text and display it."

[1674] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1675] Step 1:

[1676] A user performs a sign language motion. The camera in a smartphone or smart glasses captures the motion. The input is the user's sign language motion, and the output is the captured video data. In concrete terms, the camera continuously shoots video and captures the sign language motion in real time.

[1677] Step 2:

[1678] The server receives the captured video data and performs preprocessing. The input is the captured video data, and the output is the preprocessed image data. The specific operations include resizing the video data, removing noise, and extracting the sign language motion features.

[1679] Step 3:

[1680] The server inputs the preprocessed image data into the generative AI model to analyze the sign language motion. The input is the preprocessed image data, and the output is the analysis result, a text representation of the sign language. In concrete terms, the generative AI model analyzes the characteristics of the sign language motion and generates the corresponding text.

[1681] Step 4:

[1682] The server converts the results of the analysis into another sign language or text. The input is a textual representation of the sign language, and the output is the converted sign language or text. In particular, a translation algorithm converts the textual representation of the sign language into a different sign language or text.

[1683] Step 5:

[1684] The server sends the converted sign language or text to the terminal and displays it. The input is the converted sign language or text, and the output is the information displayed on the terminal's display. The specific operation is to visually display the converted sign language or text on the terminal's display.

[1685] Step 6:

[1686] A user speaks using voice. The microphone in a smartphone or smart glasses captures the voice. The input is the user's voice, and the output is the captured voice data. In concrete terms, the microphone continuously records the voice and captures the voice data in real time.

[1687] Step 7:

[1688] The server receives the captured voice data and performs speech recognition. The input is the captured voice data, and the output is text data that is the result of speech recognition. In concrete terms, the speech recognition algorithm analyzes the voice data and generates corresponding text.

[1689] Step 8:

[1690] The server sends the speech recognition results to the terminal and displays them. The input is the text data of the speech recognition results, and the output is the information displayed on the terminal's display. The specific operation is to visually display the speech recognition results on the terminal's display.

[1691] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1692] "Example 1"

[1693] In one embodiment of the present invention, an emotion engine is incorporated. The emotion engine recognizes the user's emotion from the captured sign motion. Specifically, the emotion engine determines the likelihood that the user is expressing an emotion such as joy, anger, sadness, surprise, etc. from the speed, strength, and rhythm of the sign motion. The emotion engine adjusts the sign-to-text conversion according to the emotion. For example, if it is determined that the user is expressing an emotion of anger, an expression reflecting that emotion can be included in the text.

[1694] "Example 2"

[1695] In another embodiment of the present invention, an emotion engine is incorporated to recognize the emotion of the user from the captured sign motion and adjust the conversion from Japanese Sign Language to International Sign Language based on the emotion. Specifically, if it is determined that the user is expressing an emotion of joy, an expression reflecting the emotion can be included in the international sign language.

[1696] The process flow of each embodiment will be described below.

[1697] "Example 1"

[1698] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[1699] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[1700] Step 3: The emotion engine adjusts the sign-to-text conversion based on the determined emotion. For example, if it determines that the user is expressing an emotion of anger, it can include a wording that reflects that emotion in the text.

[1701] "Example 2"

[1702] Step 1: Step 1: The system captures sign language motions from the user using a sensor such as a camera.

[1703] Step 2: The captured sign language motion data is sent to the emotion engine, which determines the emotion the user may be expressing based on the speed, strength, and rhythm of the motion.

[1704] Step 3: The emotion engine adjusts the conversion from Japanese Sign Language to international sign language based on the determined emotion. For example, if it determines that the user is expressing the emotion of joy, an expression reflecting that emotion can be included in the international sign language.

[1705] Example 1

[1706] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1707] Conventional sign language translation systems were able to capture sign language motions and convert them into other sign languages ​​or text, but they lacked the ability to recognize the user's emotions and adjust the translation results accordingly. This made it difficult to accurately convey the nuances and emotions of sign language, resulting in a decline in the quality of communication.

[1708] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1709] In this invention, the server includes a means for capturing a sign language motion, a means for converting the captured sign language motion into another sign language or text, a means for displaying the converted sign language or text, a means for recognizing a user's emotion from the captured sign language motion, and a means for adjusting the conversion from sign language to text in accordance with the recognized emotion. This enables sign language translation that reflects the user's emotion, realizing more natural and accurate communication.

[1710] A "means for capturing sign language motion" is a device or method that uses a camera or motion sensor to capture a user's sign language movements in real time and acquire them as digital data.

[1711] "Means for converting captured sign language motion into another sign language or text" refers to a device or method that uses an AI analysis engine to analyze acquired sign language motion data and convert it into a corresponding other sign language or text.

[1712] "Means for displaying the converted sign language or text" refers to a device or method, such as a display or monitor, for visually presenting the converted sign language or text to a user.

[1713] The "means for recognizing a user's emotions from the captured sign language motion" refers to an emotion engine or algorithm that analyzes characteristics of the sign language motion, such as speed, strength, and rhythm, and estimates the user's emotions.

[1714] "Means for adjusting the sign language-to-text conversion in response to recognized emotions" refers to an apparatus or method that adjusts the sign language-to-text conversion result based on the user's emotions recognized by an emotion engine, and generates text that reflects the emotions.

[1715] The system of the present invention includes a camera and a motion sensor for capturing the motion of the sign language, an AI analysis engine for analyzing the captured motion data, an emotion engine for recognizing the emotion of the user, and a display for displaying the converted sign language and text. Specific embodiments are described below.

[1716] Hardware and Software Used

[1717] Hardware: High resolution camera, motion sensor, display

[1718] Software: AI analysis engine, emotion engine, sign language conversion algorithm

[1719] Data Import

[1720] When a user signs, the device's camera and motion sensor capture the sign language motion in real time. The camera captures hand movements in high resolution, and the motion sensor records the hand position and movement in detail. The captured motion data is then sent from the device to a server.

[1721] 4. Data Analysis

[1722] The server inputs the received motion data into an AI analysis engine, which uses a deep learning model to analyze the motion data and convert it into the corresponding sign language or text, for example converting "hello" in Japanese sign language to "hello" in international sign language.

[1723] emotion recognition

[1724] The server inputs the analyzed sign language data into the emotion engine. The emotion engine extracts features of the sign language motion, such as speed, strength, and rhythm, to recognize the user's emotion. For example, if the sign language is fast and strong, it determines that the user is expressing the emotion of anger. It adjusts the conversion from sign language to text according to the recognized emotion. For example, if the user is expressing the emotion of joy, it generates the text "Hello!" that reflects that emotion.

[1725] Data Output

[1726] The server sends the converted sign language or text to the terminal, which displays the received data to the user. For example, the text "Hello!" is displayed on the terminal screen.

[1727] Examples

[1728] For example, when a user wants to say "hello" in Japanese Sign Language, the following specific actions are performed:

[1729] 1. The user signs.

[1730] 2. The device's camera and motion sensors capture the user's sign language motions in real time.

[1731] 3. The device transmits the captured motion data to the server.

[1732] 4. The server inputs the received motion data into the AI ​​analysis engine.

[1733] 5. The AI ​​analysis engine analyzes the motion data and converts the Japanese sign language "hello" into the international sign language "hello."

[1734] 6. The server inputs the analyzed sign language data into the emotion engine.

[1735] 7. The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[1736] 8. The emotion engine determines that the user is expressing the emotion of happiness and generates text that reflects that emotion: "Hello!"

[1737] 9. The server sends the converted sign language or text to the terminal.

[1738] 10. The device displays the received data to the user, for example displaying the text "Hello!" on the device screen.

[1739] Examples of prompt statements

[1740] Examples of prompt sentences to be input to a generative AI model include the following:

[1741] "Analyze the sign language motion data captured by the camera and convert the Japanese sign language "Hello" into the international sign language "Hello." Also, recognize the user's emotions, and if the emotion includes joy, display "Hello!" in the text."

[1742] In this way, the system can analyze sign language motions in real time and convert them into appropriate sign language and text.

[1743] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1744] Step 1:

[1745] The user signs.

[1746] As a specific action, the user says "hello" in Japanese Sign Language.

[1747] Input: User's sign language motion

[1748] Output: None

[1749] Step 2:

[1750] The device's camera and motion sensors capture the user's sign language movements in real time.

[1751] Specifically, the camera captures hand movements in high resolution, while the motion sensor records the hand's position and movement in detail.

[1752] Input: User's sign language motion

[1753] Output: Motion data

[1754] Step 3:

[1755] The terminal transmits the captured motion data to the server.

[1756] Specifically, data is transmitted in real time, minimizing latency.

[1757] Input: Motion data

[1758] Output: Motion data sent to the server

[1759] Step 4:

[1760] The server inputs the received motion data into an AI analysis engine.

[1761] Specifically, the AI ​​analytics engine uses deep learning models to analyze motion data.

[1762] Input: Motion data sent to the server

[1763] Output: Analysis results (sign language and text)

[1764] Step 5:

[1765] The server inputs the parsed sign language data into an emotion engine.

[1766] As a specific action, the emotion engine extracts features such as speed, strength and rhythm of sign language motion.

[1767] Input: Analysis results (sign language or text)

[1768] Output: Emotion data

[1769] Step 6:

[1770] The emotion engine recognizes the user's emotions from the speed, strength, rhythm, etc. of sign language motion.

[1771] Specifically, if the sign language is fast and forceful, it is determined that the user is expressing anger.

[1772] Input: Emotion data

[1773] Output: Recognized emotion

[1774] Step 7:

[1775] The emotion engine adjusts the sign language to text conversion depending on the recognized emotion.

[1776] Specifically, if the user expresses an emotion of joy, the system generates text that reflects that emotion: "Hello!"

[1777] Input: Recognized emotion

[1778] Output: Adjusted text

[1779] Step 8:

[1780] The server sends the converted sign language or text to the terminal.

[1781] Specifically, the data is encrypted and transmitted securely.

[1782] Input: Adjusted text

[1783] Output: Text sent to the terminal.

[1784] Step 9:

[1785] The terminal displays the received data to the user.

[1786] The specific action is that the text "Hello!" will be displayed on the device screen.

[1787] Input: Text sent to the terminal

[1788] Output: The displayed text

[1789] (Application example 1)

[1790] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1791] There is an issue that it is difficult for hearing-impaired people and hearing-non-hearing people to communicate smoothly in physical stores. In particular, it is difficult to convey information using sign language to hearing-nonhearing people who cannot understand sign language. In addition, there is a lack of technology that can recognize the user's emotions from the motion of sign language and convert it into text or voice that appropriately reflects those emotions, which makes it difficult to communicate accurately with emotions.

[1792] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1793] In this invention, the server includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotion from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotion, and means for reading out the converted text aloud. This enables hearing-impaired people and hearing-normal people to communicate smoothly in physical stores.

[1794] "Sign language motion" refers to the movements of the hands, arms, fingers, etc. when using sign language.

[1795] "Means of capturing" refers to devices or methods that use cameras, sensors, etc. to capture sign language motion as digital data.

[1796] A "conversion method" refers to software or algorithms used to convert captured sign motion into another sign language or text.

[1797] "Display device" means a display or screen for visually presenting the converted sign language or text.

[1798] "Means of recognizing emotions" refers to software or algorithms for analyzing and recognizing a user's emotions from sign language motions.

[1799] "Means for tailoring the conversion" means software or algorithms that appropriately adjust the conversion of sign language to text based on recognized emotions.

[1800] "A reading means" refers to a speaker or speech synthesis software for outputting the converted text as speech.

[1801] A system for implementing the present invention includes means for capturing sign language motion, means for converting the captured sign language motion into another sign language or text, means for displaying the converted sign language or text, means for recognizing a user's emotions from the captured sign language motion, means for adjusting the conversion from sign language to text to reflect the user's emotions, and means for reading aloud the converted text.

[1802] Hardware and Software Configuration

[1803] Hardware

[1804] Smart glasses: Equipped with cameras and sensors to capture sign language motions.

[1805] Computer: Built into the smart glasses, it processes data.

[1806] software

[1807] OpenCV: For capturing and pre-processing camera images.

[1808] Keras: Runs AI models for sign language and emotion recognition.

[1809] pyttsx3: Used as a text-to-speech engine.

[1810] Data processing and calculation

[1811] Sign language motion capture

[1812] We use the camera and sensors installed in the smart glasses to capture the user's sign language motion in real time. We use OpenCV to acquire and pre-process the video.

[1813] Sign language motion conversion

[1814] The preprocessed video data is fed into a Keras-based sign language recognition model to convert the sign language motion into text. It is also possible to convert it into another sign language (e.g., from Japanese Sign Language to International Sign Language).

[1815] Emotion recognition

[1816] The same video data is then fed into an emotion recognition model using Keras to analyze the user's emotions. The model recognizes emotions from the speed, strength, and rhythm of the sign language motion.

[1817] Text Adjustment

[1818] Adjust the sign-to-text conversion based on the recognized emotion. For example, if the user is expressing anger, include a statement in the text that reflects that emotion.

[1819] View and read text aloud

[1820] The converted text is displayed on the smart glasses display and then read aloud using pyttsx3.

[1821] Examples

[1822] When a hearing-impaired person uses sign language to ask a store clerk, "Where is this item?", the smart glasses will recognize the sign language and tell the store clerk, "Where is this item?" Furthermore, the emotion engine will recognize that the user is in a hurry based on the speed and strength of the sign language, and add, "You seem to be in a hurry."

[1823] Examples of prompt statements

[1824] The customer signs, "Where is this item?". From the speed and force of the sign, you recognize that the customer is in a hurry, so you verbally tell the store associate, "Where is this item? It seems like you're in a hurry."

[1825] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1826] Step 1:

[1827] A camera mounted on the smart glasses captures the user's sign language motion in real time. The input is the camera image, and the output is the captured image data. The camera i...

Claims

1. a means for capturing a sign language motion of a user; A means for generating a prompt sentence for input to a generative AI model based on the captured sign language motion; A means for inputting the generated prompt sentence into the generative AI model and obtaining a result of converting the sign language motion captured from the generative AI model into another expression; A means for displaying the obtained conversion result; A system including:

2. The system of claim 1 , further comprising means for analyzing the user's emotion using an emotion engine and adjusting a conversion result based on the analyzed user's emotion.

3. The system of claim 1 , wherein the generative AI model converts captured sign language motions into sign language or text appropriate for different countries.

Images