system

Abstract

To provide a system which enables a visually impaired or hearing-impaired person to sufficiently receive information of live coverage when watching sports.SOLUTION: A system includes: means for acquiring audio information of live coverage in watching sports; means for generating a first prompt sentence instructing translation of the acquired audio information into text information; means for generating the text information corresponding to the acquired audio information using the generated first prompt sentence and a generative AI model; means for generating a second prompt sentence instructing a translation of the generated text information into sign language data or braille data; means for generating the sign language data or the braille data corresponding to the generated text information using the generated second prompt sentence and the generative AI model; means for visually displaying the generated sign language data using sign language animation software; and means for outputting the generated braille data onto a braille display.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When visually or hearing-impaired persons watch sports games, there is a problem that they cannot fully enjoy the information of the live broadcast.

Means for Solving the Problems

[0005] As a means for solving this problem, a system is provided that captures live broadcast voice data, analyzes it with a generation AI, and converts it into sign language or braille in real time. Further, the generation AI generates information for visually or hearing-impaired persons to enjoy sports games based on the information of the live broadcast voice data, and outputs the information in real time.

Brief Description of the Drawings

[0006] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1 of Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 of Embodiment 2. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2. [Figure 15] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 3 of Example 3. [Figure 16] This is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3. [Figure 17] It is a sequence diagram showing the processing flow of the data processing system in Example 1 of Form Example 1 when the emotion engine is combined. [Figure 18] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1 when the emotion engine is combined. [Figure 19] It is a sequence diagram showing the processing flow of the data processing system in Example 2 of Form Example 2 when the emotion engine is combined. [Figure 20] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2 when the emotion engine is combined. [Figure 21] It is a sequence diagram showing the processing flow of the data processing system in Example 3 of Form Example 3 when the emotion engine is combined. [Figure 22] It is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3 when the emotion engine is combined.

Embodiments for Carrying out the Invention

[0007] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the language used in the following description will be explained.

[0009] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)), etc.

[0010] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0011] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0012] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0014] [First Embodiment]

[0015] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0016] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0027] "Example of form 1"

[0028] One embodiment of the present invention includes a microphone and audio input device for capturing audio data from live sports broadcasts. This audio input device can capture audio data from live broadcasts in real time. The captured audio data is transmitted to a generating AI.

[0029] "Example of form 2"

[0030] The generation AI analyzes the captured audio data and converts it into sign language or Braille in real time.

[0031] Specifically, it generates sign language and Braille data based on information obtained from audio data. This generation AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[0032] "Example of form 3"

[0033] The generated sign language and braille data are provided to the user through an output device. Possible output devices include displays for showing sign language and braille displays for outputting braille. For example, a braille display physically forms braille based on the generated braille data, allowing visually impaired individuals to read it by touch.

[0034] The following describes the processing flow for each example of the form.

[0035] "Example of form 1"

[0036] Step 1: Install a microphone or audio input device to capture audio data from live sports broadcasts. This audio input device can capture audio data from live broadcasts in real time.

[0037] Step 2: The captured audio data is sent to the generating AI. This generating AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[0038] "Example of form 2"

[0039] Step 1: The generation AI analyzes the captured audio data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on the information obtained from the audio data.

[0040] Step 2: The generated sign language and braille data are provided to the user through an output device. Possible output devices include displays for showing sign language and braille displays for outputting braille.

[0041] "Example of form 3"

[0042] Step 1: As a concrete example, a braille display physically forms braille based on generated braille data, which can be read by visually impaired people through touch.

[0043] (Example 1)

[0044] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0045] There is a problem in that people with visual or hearing impairments have difficulty enjoying watching sports. In particular, they have difficulty understanding the audio information of live broadcasts in real time, which greatly diminishes their enjoyment of watching sports.

[0046] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for transmitting the live broadcast audio data to a generating AI model, and means for the generating AI model to analyze the audio data and convert it into text. This makes it possible for people with visual or hearing impairments to enjoy watching sports in real time.

[0047] "Live commentary audio data" refers to audio information provided in real time by commentators during events such as sports broadcasts.

[0048] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates a specific output.

[0049] "Means" refer to the methods or devices used to achieve a specific purpose.

[0050] "Converting to text" is the process of analyzing audio data and converting it into written information.

[0051] "Outputting" means providing processed data to the user in the form of display, audio, or other formats.

[0052] Modes for carrying out the invention

[0053] This invention is a system that captures audio data from live sports broadcasts in real time and transmits that data to a generating AI model. Specifically, it acquires audio data using a microphone or voice input device and transmits that data to a generating AI model to perform various data processing and calculations.

[0054] Hardware and software to use

[0055] Hardware: Microphone, audio input device

[0056] Software: Generative AI models (e.g., GPT-4® from OpenAI®)

[0057] System Overview

[0058] 1. Importing audio data:

[0059] Users provide live commentary on sports events.

[0060] Microphones and audio input devices connected to the terminal capture audio data from the live broadcast in real time.

[0061] 2. Sending audio data:

[0062] The captured audio data is sent from the terminal to the server.

[0063] The server sends the received audio data to the generating AI model.

[0064] 3. Data analysis and processing:

[0065] The generative AI model analyzes the received audio data and performs necessary data processing and calculations.

[0066] For example, it can perform tasks such as converting audio data to text, extracting important events, and generating real-time highlights.

[0067] 4. Output of results:

[0068] The server receives the results from the generated AI model and provides feedback to the user.

[0069] Specifically, the generated text is sent to the user's device, and the device displays that text.

[0070] Specific example

[0071] Specific example 1:

[0072] A user is providing live commentary on a soccer match.

[0073] The microphone picks up the voice saying "Goal!".

[0074] The device sends this audio data to the server.

[0075] The server sends audio data to the AI ​​model, which then generates the text "Goal scored!".

[0076] Specific example 2:

[0077] A user is providing live commentary on a basketball game.

[0078] The microphone picks up the voice saying, "Three-point shot successful!"

[0079] The device sends this audio data to the server.

[0080] The server sends audio data to the AI ​​model, which generates the text "You made a three-point shot!".

[0081] Example of a prompt

[0082] Prompt message 1:

[0083] "Analyze this audio data and extract the important events. Audio data: 'Goal!'"

[0084] Prompt message 2:

[0085] "Please convert this audio data to text. Audio data: 'Three-point shot successful!'"

[0086] The above describes the embodiments for carrying out this invention.

[0087] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0088] Step 1:

[0089] The user provides live commentary on a sports event. A microphone and audio input device connected to the terminal capture the commentary audio data in real time. Specifically, when the user shouts "Goal!", the microphone captures the audio and converts it into digital data. The input is the user's voice, and the output is digital audio data.

[0090] Step 2:

[0091] The terminal sends the captured audio data to the server. Specifically, the terminal compresses the audio data and sends it to the server using the TCP / IP protocol. The input is digital audio data, and the output is the audio data sent to the server.

[0092] Step 3:

[0093] The server sends the received audio data to the generating AI model. Specifically, the server sends the audio data to the generating AI model's API and waits for the analysis results. The input is the audio data sent to the server, and the output is the audio data sent to the generating AI model.

[0094] Step 4:

[0095] The generative AI model analyzes the audio data and performs necessary data processing and calculations. Specifically, the generative AI model converts the audio data into text and generates the text "Goal scored!". The input is the audio data sent to the generative AI model, and the output is the generated text.

[0096] Step 5:

[0097] The server receives the results from the generated AI model and provides feedback to the user. Specifically, the server sends the generated text to the user's device, and the device displays that text. The input is the generated text, and the output is the text displayed on the user's device.

[0098] (Application Example 1)

[0099] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0100] In traditional sports viewing, it has been difficult for people with visual or hearing impairments to follow the progress of a match or to follow important highlights in real time. Furthermore, even for general spectators, there were limited means to obtain real-time information on important highlights and match statistics. This resulted in a limitation in the enjoyment of watching sports.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means for acquiring live commentary audio data, means for analyzing the live commentary audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live commentary audio data into text, means for generating important highlight scenes and statistical information about the match based on the transcribed commentary content, and means for outputting the generated highlight scenes and statistical information. This makes it possible for people with visual or hearing impairments to obtain information in real time to enjoy watching sports, and also allows general spectators to obtain important highlight scenes and statistical information about the match in real time.

[0103] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports and events.

[0104] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate a specific output.

[0105] Sign language is a language that uses hand and finger movements to communicate with people who are deaf or hard of hearing.

[0106] Braille is a tactile representation of writing that uses raised dots to allow visually impaired people to read text.

[0107] "Methods for converting to text" refer to technologies that analyze audio data and convert it into corresponding textual information.

[0108] A "highlight" refers to an important moment or event in a sport or event.

[0109] "Statistical information" refers to numerical data and analysis results related to matches and events.

[0110] "Means of output" refer to technologies such as display devices and audio devices that provide the generated information to the user.

[0111] A system for carrying out this invention includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the transcribed live broadcast content, and means for outputting the generated highlights and statistical information.

[0112] Hardware and software to be used

[0113] Hardware: Smartphone (with built-in microphone)

[0114] Software: Python, SpeechRecognition library (speech recognition), OpenAI API (generative AI)

[0115] Processing flow

[0116] 1. Importing audio data:

[0117] The smartphone's microphone is used to capture audio data from the live broadcast in real time. The user launches the smartphone application and begins voice input.

[0118] 2. Text conversion of audio data:

[0119] The captured audio data is converted to text using the SpeechRecognition library. The server sends the audio data to the Google® Speech Recognition API and retrieves the corresponding text data.

[0120] 3. Analysis using Generative AI:

[0121] The transcribed commentary is sent to a generating AI using the OpenAI API. The generating AI analyzes the input text data and generates important highlights and statistics.

[0122] 4. Information Output:

[0123] The generated highlights and statistics are displayed on the user's smartphone in real time. This allows users to stay informed about important moments and statistics from the match in real time.

[0124] Specific example

[0125] When a user is watching a soccer match, they launch a smartphone app to capture the audio from the live broadcast. The app converts the audio into text and displays important events such as "Goal!" and "Foul!" in real time. Furthermore, it automatically generates and displays match highlights and statistics (e.g., number of shots, ball possession).

[0126] Example of a prompt

[0127] Please generate the following sports commentary highlights:

[0128] "Team A scored a goal in the 30th minute of the first half. Team B's defense broke down. Team A continued their offensive in the second half as well."

[0129] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0130] Step 1:

[0131] The user launches a smartphone application to capture audio from a live broadcast. The user uses the smartphone's microphone to collect audio data in real time. The input is the audio data from the live broadcast, and the output is a stream of audio data.

[0132] Step 2:

[0133] The device converts the captured audio data into text using the SpeechRecognition library. The device then sends the audio data to the Google Speech Recognition API to retrieve the corresponding text data. The input is a stream of audio data, and the output is text data.

[0134] Step 3:

[0135] The server sends the transcribed commentary to the generating AI using the OpenAI API. The server analyzes the input text data and generates important highlights and statistics. The input is text data, and the output is the generated highlights and statistics.

[0136] Step 4:

[0137] The server sends the generated highlights and statistics to the user's smartphone in real time. The user can view this information on their smartphone screen. The input is the generated highlights and statistics, and the output is the information displayed on the smartphone screen.

[0138] Step 5:

[0139] Users can access important moments and statistics from a match in real time through a smartphone application. Specifically, users check the information on the screen and understand the progress of the match. The input is the information displayed on the smartphone screen, and the output is the user's understanding.

[0140] (Example 2)

[0141] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0142] There is a problem in that people with visual or hearing impairments have difficulty understanding auditory information in real time. In particular, the accuracy and speed of converting auditory information into sign language or Braille are insufficient, which can lead to delays in information transmission and misunderstandings. This creates a challenge in that it is difficult for people with visual or hearing impairments to receive information accurately and quickly.

[0143] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0144] In this invention, the server includes means for acquiring audio data, means for converting the audio data into text data using speech recognition software, and means for inputting the text data into a generation AI model to generate sign language data or Braille data. This makes it possible for visually and hearing impaired people to receive audio information in real time as sign language or Braille.

[0145] "Audio data" refers to data that records audio in digital format.

[0146] "Speech recognition software" is software that analyzes speech data and converts it into text data.

[0147] "Text data" refers to data that records character information in a digital format.

[0148] A "generative AI model" is an artificial intelligence model that is trained using deep learning technology to generate sign language data or Braille data based on text data.

[0149] "Sign language data" refers to data that records sign language movements in digital format.

[0150] "Braille data" refers to data that records Braille information in digital format.

[0151] "Sign language animation software" is software that visually displays sign language movements based on sign language data.

[0152] A "braille display" is a device that physically displays braille based on braille data.

[0153] This invention is a system that enables people with visual or hearing impairments to receive audio information in real time as sign language or Braille. Specific embodiments of this system are described below.

[0154] System Configuration

[0155] Server Configuration

[0156] The server includes means for acquiring audio data, means for converting audio data into text data using speech recognition software, and means for converting text data into sign language data or Braille data using a generative AI model. The speech recognition software used is a general speech recognition API (e.g., Google Cloud Speech-to-Text API). The generative AI model is a model trained using deep learning techniques (e.g., a model using TENSORFLOW® or PyTorch).

[0157] Device configuration

[0158] The terminal includes means for visually displaying sign language data transmitted from the server using sign language animation software (e.g., SignAll), and means for outputting braille data to a braille display (e.g., Focus Blue). The terminal also includes a microphone for the user to input voice data.

[0159] User actions

[0160] The user inputs voice data by speaking into the device's microphone. For example, they might say "hello." The device sends this voice data to a server. The server converts the voice data into text data and uses a generative AI model to generate sign language data or Braille data. The generated data is sent to the device and displayed as a sign language animation on the screen or output to a Braille display.

[0161] Specific example

[0162] Example 1: Conversion to sign language

[0163] When a user says "Good morning," the device sends the audio data to a server. The server converts the audio data into text and inputs the text information "Good morning" into a generative AI model. The generative AI model generates sign language data corresponding to "Good morning" and creates a sign language animation using sign language animation software. The device displays this sign language animation on its screen, allowing the user to visually confirm the sign language for "Good morning."

[0164] Example 2: Conversion to Braille

[0165] When the user says "thank you," the device sends the audio data to a server. The server converts the audio data into text and inputs the text information "thank you" into a generating AI model. The generating AI model generates braille data corresponding to "thank you" and sends it to a braille display. The braille display connected to the device displays "thank you" in braille, and the user can read the braille by touch.

[0166] Example of a prompt

[0167] "Please explain how the generative AI model translates a user's 'Good morning' into sign language or Braille."

[0168] By inputting this prompt into the generating AI model, a detailed explanation of the specific transformation process can be obtained.

[0169] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0170] Step 1:

[0171] The user inputs voice data.

[0172] The user speaks into the device's microphone. For example, they might say "Hello." The device captures this voice as digital audio data. The input is the user's voice, and the output is digital audio data.

[0173] Step 2:

[0174] The device sends voice data to the server.

[0175] The device transmits the captured audio data to the server via the internet. During this process, the data is encoded in an appropriate format (e.g., WAV or MP3). The input is digital audio data, and the output is the audio data transmitted to the server.

[0176] Step 3:

[0177] The server converts the audio data into text data.

[0178] The server converts the received audio data into text data using speech recognition software (e.g., a common speech recognition API). For example, the audio data "Hello" is converted into the text data "Hello". The input is audio data, and the output is text data.

[0179] Step 4:

[0180] The server generates text data and inputs it into the AI ​​model.

[0181] The server inputs the converted text data into a generative AI model. This generative AI model is trained using deep learning techniques. The input is text data, and the output is sign language data or Braille data.

[0182] Step 5:

[0183] The generative AI model generates sign language data and Braille data.

[0184] The generative AI model generates sign language data and Braille data based on input text data. For example, the text data "Hello" is converted into sign language animation data and Braille data. The input is text data, and the output is sign language data and Braille data.

[0185] Step 6:

[0186] The server sends the generated data to the terminal.

[0187] The server sends the generated sign language or braille data to the terminal. The data is encoded in an appropriate format (e.g., JSON or XML). The input is the sign language or braille data, and the output is the data sent to the terminal.

[0188] Step 7:

[0189] The terminal displays sign language animations and outputs Braille data to a Braille display.

[0190] The terminal displays received sign language data on its screen using sign language animation software (e.g., sign language animation software). Braille data is output to a braille display connected to the terminal (e.g., a braille display). The user can visually confirm the sign language animation displayed on the screen and tactilely read the braille displayed on the braille display. Input is sign language data and braille data, and output is sign language animation and braille.

[0191] (Application Example 2)

[0192] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0193] There is a problem in that visually impaired and hearing-impaired individuals have difficulty understanding voice instructions and warnings in real time during security work. This can prevent them from performing their duties efficiently, potentially leading to a decline in the quality of security.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the audio data into text, means for converting the text into sign language and Braille, and means for providing the sign language and Braille to visually impaired and hearing impaired people. This makes it possible for visually impaired and hearing impaired people to understand voice instructions and warnings in real time and perform security work efficiently.

[0195] "Live broadcast audio data" refers to audio information of events and situations that are taking place in real time.

[0196] "Generative AI" refers to artificial intelligence technology that analyzes audio data and converts it into sign language or Braille.

[0197] "Sign language" refers to a means of conveying linguistic information using visual gestures and movements.

[0198] Braille refers to a writing system that uses raised dots to allow visually impaired people to read text information using their sense of touch.

[0199] "Audio data" refers to information recorded in digital format using audio.

[0200] "Text" refers to written information.

[0201] "Visually impaired" refers to people who have a visual impairment.

[0202] "People with hearing impairments" refers to individuals who have a hearing loss.

[0203] "Security work" refers to work performed to protect the safety of facilities and people.

[0204] "Real-time" refers to processing or communication occurring instantly without delay.

[0205] The system for carrying out this invention converts audio data into sign language or Braille in real time and provides it to visually impaired and hearing impaired individuals. Specific embodiments of the system are described below.

[0206] System Configuration

[0207] hardware

[0208] Smartphone: Equipped with a microphone for acquiring audio data and a display for showing the conversion results.

[0209] Server: Provides computing resources for analyzing and converting audio data.

[0210] software

[0211] Speech recognition library: Use the speech_recognition library to convert speech data to text.

[0212] Sign language conversion library: Use the hand_gesture_recognition library to convert text into sign language.

[0213] Braille conversion library: Use the braille_converter library to convert text to Braille.

[0214] Generative AI Model: Uses a deep learning model to analyze audio data and convert it into sign language or Braille.

[0215] System operation

[0216] Acquisition of audio data

[0217] The user inputs voice data using their smartphone's microphone. For example, a security guard might input "There is a suspicious person" by voice.

[0218] Analysis of audio data

[0219] The server converts the acquired audio data into text using the speech_recognition library. This text forms the basis for subsequent sign language and Braille conversion.

[0220] Conversion to sign language and Braille

[0221] The server uses a generative AI model to convert text into sign language and Braille. The `hand_gesture_recognition` library is used for sign language conversion, and the `braille_converter` library is used for Braille conversion.

[0222] Providing results

[0223] The converted sign language and Braille are displayed on the smartphone screen. This allows visually impaired and hearing-impaired individuals to understand voice instructions and warnings in real time.

[0224] Specific example

[0225] For example, if a security guard voice-inputs "There is a suspicious person," the system converts this voice into text, and then into sign language and Braille. The results are displayed on a smartphone screen, ensuring that the information is also conveyed to security guards who are visually impaired or hearing impaired.

[0226] Example of a prompt

[0227] Please enter the audio data. Example: "There is a suspicious person."

[0228] In this way, visually impaired and hearing-impaired individuals can understand voice instructions and warnings in real time, enabling them to perform security tasks efficiently.

[0229] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0230] Step 1:

[0231] The user inputs voice data using the smartphone's microphone. The input voice data is temporarily stored in the smartphone's internal memory.

[0232] Step 2:

[0233] The smartphone sends the stored audio data to the server. The server receives the audio data and uses the speech_recognition library to convert it to text. The input is audio data, and the output is text data.

[0234] Step 3:

[0235] The server analyzes text data using a generative AI model. This analysis converts the text data into sign language and Braille. The input is text data, and the output is sign language data and Braille data.

[0236] Step 4:

[0237] The server converts sign language data into sign language using the hand_gesture_recognition library. The input is text data, and the output is sign language data.

[0238] Step 5:

[0239] The server converts braille data into braille using the braille_converter library. The input is text data, and the output is braille data.

[0240] Step 6:

[0241] The server sends the converted sign language and Braille data to the smartphone. The smartphone displays the received sign language and Braille data on its screen. This allows visually impaired and hearing-impaired individuals to understand voice instructions and warnings in real time.

[0242] (Example 3)

[0243] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0244] The aim is to solve the problem that users with visual or hearing impairments have difficulty converting text data into sign language or Braille and accessing information in real time. In particular, there is a need for a system that can quickly and accurately convert text data entered by users into sign language or Braille and provide it through an appropriate output device.

[0245] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0246] In this invention, the server includes means for a user to input text data, means for the server to receive the input data and send prompt sentences to a generating AI model, means for the generating AI model to generate sign language or Braille data, means for the server to transmit the generated data to an output device, and means for the terminal to provide sign language or Braille to the user through the output device. This makes it possible for users with visual or hearing impairments to quickly and accurately convert the input text data into sign language or Braille and use the information in real time.

[0247] A "user" refers to a person who uses the system to input text data and receives the information converted into sign language or Braille.

[0248] A "server" refers to a computer system that receives input data from users, sends prompt messages to the AI ​​generation model, and transmits the generated sign language or Braille data to an output device.

[0249] A "generative AI model" refers to an artificial intelligence model that generates sign language or Braille data based on text data from users.

[0250] A "prompt sentence" refers to a sentence used to instruct a generative AI model to convert text data into sign language or Braille.

[0251] An "output device" is a device that provides generated sign language or Braille data to the user, and includes sign language displays and Braille displays.

[0252] A "sign language display" refers to a display device used to visually display generated sign language data.

[0253] A "braille display" refers to a device that physically forms generated braille data, allowing visually impaired individuals to read it by touch.

[0254] "Text data" refers to the character information that a user enters into the system.

[0255] "Real-time" refers to the process where text data entered by a user is instantly converted into sign language or Braille and provided almost simultaneously through an output device.

[0256] Modes for carrying out the invention

[0257] This invention is a system that enables users with visual or hearing impairments to convert text data into sign language or Braille and access information in real time. Specific embodiments of this system are described below.

[0258] System Configuration

[0259] This system consists of the following main components:

[0260] 1. User terminal: A device used by a user to input text data. Examples include personal computers, smartphones, and tablets.

[0261] 2. Server: This server receives input data from the user, sends prompt messages to the generation AI model, and sends the generated sign language or Braille data to the output device. A high-performance server (e.g., a server equipped with an NVIDIA GPU) should be used.

[0262] 3. Generative AI Model: An artificial intelligence model for generating sign language or Braille data based on text data from users. It uses deep learning frameworks such as TensorFlow or PyTorch.

[0263] 4. Output device: A device for providing the generated sign language or Braille data to the user. This includes sign language displays and Braille displays.

[0264] System operation

[0265] 1. The user enters text data.

[0266] Users input text data through the system interface. For example, they might type "Hello" using a web browser or a dedicated application.

[0267] 2. The server receives the input data and sends a prompt message to the generating AI model.

[0268] The server receives text data sent by the user. Based on the received data, it creates and sends a prompt to the generative AI model. For example, it might send the prompt "Please convert the following text into sign language: Hello" to the generative AI model.

[0269] 3. The generation AI model generates sign language and braille data.

[0270] The generation AI model analyzes the prompt sentence received from the server and generates corresponding sign language and braille data. For example, it generates sign language animation data or braille array data.

[0271] 4. The server sends the generated data to the output device.

[0272] The server sends the sign language and braille data received from the generation AI model to the output device. For example, it sends sign language animation data to a sign language display and braille data to a braille display.

[0273] 5. The terminal provides sign language and braille to the user through the output device.

[0274] The terminal receives the data sent from the server and provides it to the user through the output device. The sign language display shows sign language animations, and the braille display physically forms braille. The user can visually check the sign language display and tactilely read the braille display.

[0275] Specific Example

[0276] For example, the specific operations when the user inputs the text "Thank you" are as follows.

[0277] 1. User: Input "Thank you" into the input field of the web browser and click the send button.

[0278] 2. Server: Receives the text data "Thank you" from the user and sends the prompt sentence "Please convert the following text into sign language: Thank you" to the generation AI model.

[0279] 3. Generate AI Model: Analyze the prompt text and generate sign language animation data corresponding to "Thank you".

[0280] 4. Server: Send the generated sign language animation data to the sign language display.

[0281] 5. Terminal: The sign language display shows the sign language animation, and the user visually confirms it.

[0282] In this way, the user can easily convert text into sign language or braille and use it through the output device. The flow of the specific process in Example 3 will be described using FIG. 15.

[0283] Step 1:

[0284] The user inputs text data.

[0285] The user inputs text data using a web browser or a dedicated application. For example, the user inputs "Hello" and clicks the send button. The input data is sent from the user terminal to the server.

[0286] Step 2:

[0287] The server receives the input data and sends the prompt text to the generate AI model.

[0288] The server receives the text data sent from the user. Based on the received data, a prompt text is created and sent to the generate AI model. For example, a prompt text such as "Please convert the following text into sign language: Hello" is sent to the generate AI model. The input is the user's text data, and the output is the prompt text to the generate AI model.

[0289] Step 3:

[0290] The generate AI model generates sign language or braille data.

[0291] The generative AI model analyzes prompt text received from the server and generates corresponding sign language or Braille data. For example, it can generate sign language animation data or Braille sequence data. The input is prompt text, and the output is sign language or Braille data.

[0292] Step 4:

[0293] The server sends the generated data to the output device.

[0294] The server transmits the sign language and Braille data received from the generating AI model to the output device. For example, it transmits sign language animation data to a sign language display and Braille data to a Braille display. The input is the generated sign language and Braille data, and the output is the transmission of data to the output device.

[0295] Step 5:

[0296] The terminal provides sign language and Braille to the user through an output device.

[0297] The terminal receives data transmitted from the server and provides it to the user through an output device. The sign language display shows sign language animations, and the braille display physically forms braille characters. The user can visually confirm the sign language display and tactilely read the braille display. The input is sign language or braille data from the server, and the output is the provision of sign language or braille to the user.

[0298] (Application Example 3)

[0299] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0300] There is a problem that people with visual or hearing impairments have difficulty receiving information in real time when watching sports games or obtaining security information. Also, although rapid information transmission using sign language or Braille in emergencies is required, the current system is insufficient in dealing with it.

[0301] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 3 is realized by the following respective means. In this invention, the server includes means for capturing live relay voice data, means for analyzing the live relay voice data with a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing the video of a security camera, analyzing it with a sign language recognition model, and recognizing sign language, means for displaying the recognized sign language on the video, and means for outputting security information to a Braille display. Thereby, people with visual or hearing impairments can receive sports game viewing or security information in real time.

[0302] "Live relay voice data" is data that conveys the progress of a sports event or other live event by voice.

[0303] "Generation AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[0304] "Sign language" is a visual gestural language used by people with hearing impairments for communication.

[0305] "Braille" is a character system using raised dots used by people with visual impairments for tactile reading.

[0306] "Sign language recognition model" is a machine learning model for analyzing the movements of sign language from camera video and converting them into the corresponding language.

[0307] "Security camera" is a camera device for monitoring a specific area and acquiring video data.

[0308] A "braille display" is a device that displays braille using electronically controlled raised dots.

[0309] "Security information" refers to information regarding anomalies or emergencies within the monitored area.

[0310] The system for carrying out this invention is configured as follows: The server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for acquiring video from a security camera, analyzing it with a sign language recognition model and recognizing sign language, means for displaying the recognized sign language on the video, and means for outputting security information to a Braille display.

[0311] Hardware and software to be used

[0312] Hardware:

[0313] Smartphone camera

[0314] Braille display (e.g., BrailleDisplay)

[0315] Security camera

[0316] software:

[0317] OpenCV: Acquiring and displaying camera images

[0318] HandSignRecognition: Sign Language Recognition Model

[0319] Braille Display: Braille display control library

[0320] Data processing and data calculation

[0321] 1. Importing live broadcast audio data:

[0322] The server receives audio data from live broadcasts that provide updates on the progress of sporting events and other live events.

[0323] 2. Analysis using Generative AI:

[0324] The server uses AI to analyze the captured live broadcast audio data and converts it into sign language or Braille in real time.

[0325] 3. Output of sign language and Braille:

[0326] The server outputs the generated sign language or Braille through a display or Braille display.

[0327] 4. Capture video from security cameras:

[0328] The server captures video from security cameras and analyzes it using a sign language recognition model.

[0329] 5. Recognition and display of sign language:

[0330] The server uses a sign language recognition model to recognize sign language in the video and displays the recognized sign language on the video.

[0331] 6. Outputting security information to a Braille display:

[0332] The server outputs security information to a braille display.

[0333] Specific example

[0334] Watching sports:

[0335] Users with visual or hearing impairments can receive information via displays or braille displays by having live audio data from sports events converted into sign language or braille in real time.

[0336] Obtaining security information:

[0337] Security cameras capture video footage, a sign language recognition model recognizes the sign language, and the recognized sign language is displayed on the video. Additionally, security information is output to a Braille display, allowing visually impaired individuals to receive information through touch.

[0338] Example of a prompt

[0339] "Create a program that analyzes security camera footage, displays emergency information in sign language, and outputs security information to a Braille display."

[0340] The above describes the embodiments for carrying out this invention.

[0341] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[0342] Step 1:

[0343] The server receives audio data from live broadcasts of sporting events and other live events, providing commentary on the progress of those events. The input is the live broadcast audio data, and the output is the captured audio data. This audio data is used for subsequent analysis.

[0344] Step 2:

[0345] The server analyzes the incoming live broadcast audio data using a generating AI and converts it into sign language and Braille in real time. The input is live broadcast audio data, and the output is sign language data and Braille data. The generating AI analyzes the audio data and generates the corresponding sign language and Braille.

[0346] Step 3:

[0347] The server outputs the generated sign language and Braille through a display or Braille display. Input is sign language data and Braille data, and output is sign language display on the display and Braille display on the Braille display. This allows users with visual or hearing impairments to receive information.

[0348] Step 4:

[0349] The server captures video from security cameras and analyzes it using a sign language recognition model. The input is video data from the security cameras, and the output is recognized sign language data. The sign language recognition model analyzes the video data and recognizes sign language movements.

[0350] Step 5:

[0351] The server displays the recognized sign language on the video. The input is the recognized sign language data, and the output is the sign language displayed on the video. This allows the user to understand security camera footage in sign language.

[0352] Step 6:

[0353] The server outputs security information to a braille display. The input is security information, and the output is a braille display on the braille display. This allows the user to receive security information through touch.

[0354] The above are the specific processing steps for carrying out this invention.

[0355] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0356] "Example of form 1"

[0357] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes the user's emotions and generates information to help visually or hearing-impaired individuals enjoy watching sports more. Specifically, it estimates emotions from the user's facial expressions, tone of voice, and word choice, and analyzes live broadcast audio data according to those emotions. For example, if the user is showing emotion of joy, the system recognizes that emotion and highlights more exciting scenes, converting them into sign language or Braille.

[0358] "Example of form 2"

[0359] Furthermore, the emotion engine recognizes emotions in real time and generates information accordingly. This makes it possible to provide information that allows visually and hearing-impaired individuals to enjoy watching sports in accordance with the emotions the user is feeling in real time. Specifically, if a user expresses surprise, the system recognizes that emotion, emphasizes the surprising scene, and converts it into sign language or Braille.

[0360] "Example of form 3"

[0361] Furthermore, the emotion engine generates information in response to the user's emotions and outputs that information in real time. This makes it possible to instantly provide information that allows visually and hearing-impaired individuals to enjoy watching sports, tailored to the emotions the user is feeling. Specifically, if a user expresses sadness, the system recognizes that emotion, emphasizes the sad scene, and converts it into sign language or Braille.

[0362] The following describes the processing flow for each example of the form.

[0363] "Example of form 1"

[0364] Step 1: Estimate the user's emotions from their facial expressions, tone of voice, and word choice.

[0365] Step 2: Analyze the live commentary audio data based on the estimated emotions.

[0366] Step 3: Based on the analysis results, highlight the most exciting scenes and convert them into sign language or Braille.

[0367] "Example of form 2"

[0368] Step 1: Recognize the user's emotions in real time.

[0369] Step 2: Analyze the live commentary audio data based on the recognized emotions.

[0370] Step 3: Based on the analysis results, highlight the surprising scenes and convert them into sign language or Braille.

[0371] "Example of form 3"

[0372] Step 1: Recognize the user's emotions in real time and generate information in response to those emotions.

[0373] Step 2: Output the generated information in real time.

[0374] Step 3: Based on the output information, emphasize the sad scenes and convert them into sign language or Braille.

[0375] (Example 1)

[0376] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0377] When users with visual or hearing impairments enjoy watching sports, they face the challenge of understanding the audio information in live broadcasts in real time. Furthermore, the inability to provide information that aligns with the user's emotions means that the enjoyment of watching the game is not fully conveyed.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generation AI model and recognizing the user's emotions, and means for converting the live broadcast audio data into sign language or Braille in real time according to the user's emotions. This makes it possible to provide real-time information for users with visual or hearing impairments to enjoy watching sports.

[0379] "Live commentary audio data" refers to data that transmits the progress of sporting events and other live events in real time via audio.

[0380] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to analyze data and generate information tailored to a specific purpose.

[0381] "Means of recognizing user emotions" refer to technologies and devices that analyze a user's facial expressions, tone of voice, word choice, etc., to estimate their emotional state.

[0382] "Means of converting to sign language or Braille" refer to software and hardware for converting audio data or text data into sign language videos or Braille information.

[0383] "Means of outputting sign language or Braille" refer to output devices such as displays and Braille displays that provide users with generated sign language videos and Braille information.

[0384] An "exciting scene" refers to a particularly important moment or play in a sporting event that evokes excitement or emotion.

[0385] This invention is a system for users with visual or hearing impairments to enjoy watching sports. This system captures live broadcast audio data in real time, recognizes the user's emotions, and provides information corresponding to those emotions in sign language or Braille.

[0386] Hardware and software to use

[0387] hardware

[0388] Microphones: Installed in stadiums and event venues to capture audio data of live broadcasts in real time.

[0389] Camera: Used to capture the user's facial expressions.

[0390] Terminal: A device used to process audio and video data and send it to a server.

[0391] Braille display: An output device for providing Braille information to users.

[0392] software

[0393] Generative AI model: An algorithm that analyzes voice data and recognizes the user's emotions.

[0394] Emotion engine: Software used to estimate emotions from a user's facial expressions, tone of voice, word choice, etc.

[0395] Sign language translation software: Software used to convert audio or text data into sign language videos.

[0396] Braille conversion software: Software used to convert audio data and text data into Braille information.

[0397] Specific operation of the system

[0398] Importing audio data

[0399] The device uses high-sensitivity microphones installed in the stadium to capture audio data from the live broadcast in real time. For example, it can capture the audio at the moment the commentator shouts "Goal!".

[0400] Sending audio data

[0401] The device transmits the captured audio data to a generative AI model in the cloud via the internet. Specifically, the audio data is compressed and sent to the server using a secure communication protocol.

[0402] User emotion recognition

[0403] The server monitors whether the user is connected to the system via camera and microphone. The emotion engine analyzes the user's facial expressions (e.g., smiles or expressions of surprise) and voice tone (e.g., excited voice) in real time to estimate the user's emotions.

[0404] Emotion-based data analysis

[0405] If the server estimates that the user's emotion is one of joy, it will highlight exciting scenes from the live commentary audio data. For example, it will identify and extract moments of goals or important plays.

[0406] Information generation and provision

[0407] The server converts the extracted exciting scenes into sign language and Braille. Specifically, it uses sign language translation software to generate sign language videos and Braille conversion software to display them on a Braille display. The generated information is sent to the user's terminal and displayed in real time.

[0408] Specific examples and prompt statements

[0409] Specific example

[0410] Let's say a user is watching a soccer match. If the user shows an expression of joy during a goal, the system recognizes that emotion and emphasizes the commentary about the goal, converting it into sign language or Braille.

[0411] Example of a prompt

[0412] "Please capture the live commentary audio of a goal scored during a soccer match in real time, recognize the user's emotions of joy, and convert that scene into sign language or Braille."

[0413] In this way, the system provides information tailored to the user's emotions, enabling people with visual or hearing impairments to enjoy watching sports more.

[0414] The flow of the specific processing in Example 1 will be explained using Figure 17.

[0415] Step 1:

[0416] The device uses high-sensitivity microphones installed in the stadium to capture audio data from the live broadcast in real time. The input is the audio data from the live broadcast, and the output is the captured audio data. Specifically, it uses a noise-canceling function to remove background noise.

[0417] Step 2:

[0418] The device transmits the captured audio data to a generative AI model in the cloud via the internet. The input is the captured audio data, and the output is the audio data sent to the generative AI model. Specifically, the audio data is compressed and sent to the server using a secure communication protocol.

[0419] Step 3:

[0420] The server monitors the user's connection to the system via camera and microphone. Inputs are the user's facial expression data and voice tone data, and outputs are the data sent to the emotion engine. Specifically, the camera captures the user's facial expressions, and the microphone collects the user's voice tone.

[0421] Step 4:

[0422] The server uses an emotion engine to estimate the user's emotions from their facial expressions, tone of voice, and word choice. The input is the user's facial expression data and tone of voice data, and the output is the estimated emotion data. Specifically, the emotion engine analyzes the data in real time to estimate the user's emotions.

[0423] Step 5:

[0424] If the server estimates the user's emotion to be joy, it highlights exciting scenes from the live commentary audio data. The input is the estimated emotion data and the live commentary audio data, and the output is the data of the highlighted exciting scenes. Specifically, it uses an algorithm to detect certain keywords (e.g., "goal," "great play") contained in the live commentary audio data.

[0425] Step 6:

[0426] The server converts the extracted exciting scenes into sign language or Braille. The input is data of the highlighted exciting scenes, and the output is sign language video or Braille information. Specifically, it uses sign language translation software and Braille conversion software to convert the data.

[0427] Step 7:

[0428] The server transmits the generated sign language video and Braille information to the user's terminal. The input is the sign language video and Braille information, and the output is the sign language video and Braille information displayed on the user's terminal. Specifically, a low-latency communication protocol is used to ensure real-time operation.

[0429] In this way, the system provides information tailored to the user's emotions, enabling people with visual or hearing impairments to enjoy watching sports more.

[0430] (Application Example 1)

[0431] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0432] When users with visual or hearing impairments enjoy watching sports, simply converting the audio data of live broadcasts into sign language or Braille in real time presents a challenge in providing information that aligns with the user's emotions. Furthermore, the lack of emphasis on exciting scenes that take the user's emotions into consideration makes it difficult for visually and hearing-impaired individuals to fully enjoy watching sports.

[0433] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0434] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, and means for highlighting exciting scenes according to those emotions. This makes it possible to provide information that responds to the user's emotions so that visually and hearing impaired people can enjoy watching sports more.

[0435] "Live broadcast audio data" refers to audio information that conveys the progress of sporting events and other live events in real time.

[0436] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate a specific output.

[0437] Sign language is a means of conveying linguistic information using visual gestures and movements.

[0438] Braille is a pattern of dots raised on paper or other surfaces so that visually impaired people can read it using their sense of touch.

[0439] "Means of recognizing user emotions" refers to technologies that estimate emotions from a user's facial expressions, tone of voice, word choice, etc.

[0440] "Methods for emphasizing exciting scenes" refers to techniques for selecting particularly exciting moments in sports viewing and emphasizing those moments to people with visual or hearing impairments.

[0441] The system for carrying out this invention is configured as follows: First, the server is equipped with a microphone and an audio input device for capturing live broadcast audio data. This audio input device can capture live broadcast audio data of sports events in real time.

[0442] Next, the captured audio data is sent to a generating AI. The generating AI analyzes the audio data and converts it into sign language or Braille in real time. This converted sign language or Braille data is output in a format that can be understood by people with visual or hearing impairments.

[0443] Furthermore, the system is equipped with an emotion recognition engine to recognize the user's emotions. This engine estimates emotions from the user's facial expressions, tone of voice, and word choice. The emotion recognition engine acquires and analyzes the user's emotional data using a camera and microphone.

[0444] When a user's emotions are recognized, the system highlights exciting scenes accordingly. For example, if a user expresses joy, the system recognizes this emotion and selects and highlights particularly exciting scenes. This highlighted information is then translated into sign language or Braille and provided to people with visual or hearing impairments.

[0445] The specific hardware used includes a microphone, camera, and server. The software used includes Python, the speech_recognition library, the transformers library, and the opencv library. Combining this hardware and software enables the capture of speech data, emotion recognition, conversion to sign language or Braille, and highlighting of exciting scenes.

[0446] For example, if a user expresses feelings of joy while watching a sporting event, the system recognizes those feelings and translates them into sign language or Braille, highlighting particularly exciting scenes. This information is provided to the user in real time.

[0447] Examples of prompt statements to input into a generative AI model include the following:

[0448] If a user is expressing feelings of joy, how can I highlight the exciting scenes and translate them into sign language or Braille?

[0449] This prompt can be used to query the generative AI model for specific sign language or Braille conversion methods.

[0450] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[0451] Step 1:

[0452] The server uses a microphone to capture live commentary audio data in real time.

[0453] Input: Live broadcast audio data

[0454] Output: Captured audio data

[0455] Specific operation: The server captures the audio signal from the microphone as digital data and performs preprocessing for speech recognition.

[0456] Step 2:

[0457] The server sends the captured audio data to the AI ​​for analysis.

[0458] Input: Captured audio data

[0459] Output: Analyzed audio data

[0460] Specific operation: The server inputs the audio data into a generation AI model, which then performs speech recognition and natural language processing. This converts the audio data into text data.

[0461] Step 3:

[0462] The server converts the analyzed audio data into sign language or Braille in real time.

[0463] Input: Analyzed audio data

[0464] Output: Sign language data, Braille data

[0465] Specific operation: The server executes an algorithm that converts text data into sign language or Braille using a generative AI model. The sign language data is output in video format, and the Braille data is output in a data format for tactile displays.

[0466] Step 4:

[0467] The server uses a camera to capture the user's facial expressions and sends them to the emotion recognition engine.

[0468] Input: User's facial expression data

[0469] Output: Recognized emotion data

[0470] Specific operation: The server acquires video data from the camera and inputs it into the emotion recognition engine. The emotion recognition engine performs facial expression analysis and estimates the user's emotions.

[0471] Step 5:

[0472] The server highlights exciting scenes based on recognized emotion data.

[0473] Input: Recognized emotion data, sign language data, braille data

[0474] Output: Highlighted sign language data, Highlighted braille data

[0475] Specific operation: The server analyzes emotional data and, if the user is expressing feelings of joy, selects and highlights particularly exciting scenes. This adjusts the sign language and braille data according to the user's emotions.

[0476] Step 6:

[0477] The server provides users with highlighted sign language and Braille data.

[0478] Input: Highlighted sign language data, Highlighted Braille data

[0479] Output: Sign language data and Braille data provided to the user.

[0480] Specific operation: The server sends highlighted sign language data to the user in video format and braille data in a data format for tactile displays. The user receives this data in real time and can enjoy watching sports.

[0481] (Example 2)

[0482] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0483] When users with visual or hearing impairments enjoy watching sports, they face the challenge of understanding the audio information in live broadcasts in real time. Furthermore, because information is not provided in a way that responds to the user's emotions, it is difficult to share the sense of presence and emotion.

[0484] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0485] In this invention, the server includes means for acquiring live broadcast audio data, means for preprocessing the live broadcast audio data, means for analyzing the preprocessed audio data using a generation AI model and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions from the audio data, and means for generating information based on the recognized emotions. This makes it possible for users with visual or hearing impairments to enjoy watching sports in real time.

[0486] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports events and other events.

[0487] "Preprocessing" refers to processes such as noise reduction and normalization performed to convert audio data into a format that is easier to analyze.

[0488] A "generative AI model" is an artificial intelligence model trained using deep learning technology to analyze speech data and convert it into sign language or Braille.

[0489] Sign language is a visual gesture language used by visually impaired people to understand auditory information.

[0490] Braille is a writing system that uses raised dots to allow visually impaired people to understand written information through touch.

[0491] "Emotion recognition" is an analytical technique used to identify a user's emotions from voice data.

[0492] "Information generation" is the process of creating information to provide to the user based on analyzed voice data and recognized emotions.

[0493] "Output means" refers to devices or interfaces for providing generated sign language or Braille information to the user.

[0494] This invention is a system for users with visual or hearing impairments to enjoy watching sports in real time. Specific embodiments of this system are described below.

[0495] System Configuration

[0496] hardware

[0497] The server uses a microphone (e.g., a standard microphone) to capture audio data. It also has a high-performance processor and memory for processing and analyzing the audio data. Furthermore, it is equipped with a display and a braille display for outputting sign language and braille information.

[0498] software

[0499] The server uses Librosa, a Python library, to preprocess audio data. The generative AI model is built using TensorFlow and PyTorch, and analyzes audio data to convert it into sign language or Braille. OpenCV and the Emotion API are used for emotion recognition.

[0500] System operation

[0501] The server captures live broadcast audio data in real time via a microphone. The captured audio data is preprocessed using Librosa, including noise reduction and normalization. The preprocessed audio data is then input into a generative AI model and converted into sign language or Braille.

[0502] Simultaneously, the server uses an emotion engine to recognize the user's emotions in real time from the audio data. Based on the recognized emotions, information tailored to the user's feelings is generated. For example, if the user expresses surprise, the system recognizes that emotion and highlights the surprising scene, converting it into sign language or Braille.

[0503] The generated sign language and Braille information is provided to the user via the terminal. In the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[0504] Specific example

[0505] If a user exclaims "Amazing!" while watching a sporting event, the server captures the audio and inputs it into a generative AI model. The generative AI model analyzes the audio and generates data that converts the exclamation "Amazing!" into sign language or Braille. Simultaneously, an emotion engine recognizes the user's emotion of surprise and generates information that emphasizes the surprising scene according to that emotion.

[0506] Example of a prompt

[0507] "Please describe a system that, when a user exclaims in surprise while watching a sporting event, converts that sound into sign language or Braille in real time, highlighting the surprising moment and providing information to people with visual or hearing impairments."

[0508] This system allows users with visual or hearing impairments to enjoy watching sports in real time.

[0509] The flow of the specific processing in Example 2 will be explained using Figure 19.

[0510] Step 1:

[0511] The server captures live broadcast audio data in real time via a microphone. The input is analog audio data, which is converted to a digital format and stored in temporary storage. Specifically, the audio input device captures the audio and performs digital signal processing.

[0512] Step 2:

[0513] The server preprocesses the captured audio data. The input is digital audio data, and Librosa is used for noise reduction and normalization. The output is the preprocessed audio data. Specifically, a noise filtering algorithm is applied to equalize the volume levels.

[0514] Step 3:

[0515] The server inputs pre-processed audio data into a generative AI model. The input is pre-processed audio data, which is analyzed by the generative AI model (using TensorFlow or PyTorch). The output is data for conversion into sign language or Braille. Specifically, a speech recognition algorithm analyzes the audio data and generates corresponding sign language or Braille data.

[0516] Step 4:

[0517] The server uses an emotion engine to recognize the user's emotions from audio data. The input is pre-processed audio data, and emotions are identified using OpenCV or the Emotion API. The output is the recognized emotion data. Specifically, it analyzes the tone and intonation of the voice to identify emotion categories (e.g., surprise, joy, sadness).

[0518] Step 5:

[0519] The server generates information based on recognized emotions. Input consists of emotion data and sign language or Braille data, which are combined to generate information for the user. Output is sign language or Braille information corresponding to the emotion. Specifically, it adjusts the content of the sign language or Braille based on the emotion data and sets points to emphasize.

[0520] Step 6:

[0521] The device provides users with generated sign language or Braille information. Input is sign language or Braille information corresponding to emotions, and output is information displayed on a screen or Braille display. Specifically, in the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[0522] (Application Example 2)

[0523] Next, we will describe Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0524] When people with visual or hearing impairments enjoy watching sports, they often have difficulty understanding live commentary in real time. Furthermore, because information is not provided in a way that responds to the user's emotions, there is a challenge in highlighting information that reflects their changing feelings. As a result, people with visual or hearing impairments are unable to fully enjoy watching sports.

[0525] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0526] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and converting the information with emphasis into sign language or Braille. This makes it possible for visually and hearing impaired people to enjoy watching sports in real time. Furthermore, because information is emphasized according to the user's emotions, a more immersive viewing experience can be provided.

[0527] "Live broadcast audio data" refers to audio information obtained from live broadcasts of sporting events and other live events.

[0528] "Generative AI" refers to artificial intelligence technology that analyzes audio data and converts it into sign language or Braille.

[0529] Sign language is a visual gesture language used by visually impaired people to understand auditory information.

[0530] Braille is a pattern of raised dots used by visually impaired people to understand written information through touch.

[0531] An "emotion engine" is a technology that recognizes a user's emotions in real time and generates information that corresponds to those emotions.

[0532] "User emotions" refers to the emotional states, such as surprise, joy, and sadness, that users experience while watching sports.

[0533] "Highlighting information" means displaying specific information in a way that makes it stand out, depending on the user's emotions.

[0534] The system for carrying out this invention is configured as follows: The server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and converting the information with emphasis into sign language or Braille.

[0535] Program Processing Description

[0536] The server uses the smartphone's microphone to acquire live broadcast audio data. The acquired audio data is converted to text using the speech_recognition library. Next, a generative AI model is used to convert the text data into sign language or Braille. In this process, TensorFlow is used to execute the sign language and Braille conversion model.

[0537] Furthermore, the emotion engine uses the emotion_recognition library to recognize the user's emotions in real time. Based on the recognized emotions, the generative AI emphasizes specific information and converts it into sign language or Braille. This makes it possible to provide information that is tailored to the user's emotions.

[0538] Specific example

[0539] For example, if a user is watching a live broadcast of a sporting event, the commentary audio is captured through the smartphone's microphone. The audio data is converted to text in real time, and that text can then be converted into sign language or Braille. Simultaneously, an emotion engine recognizes the user's emotions, and specific scenes or information are highlighted according to those emotions, such as surprise or joy.

[0540] Example of a prompt

[0541] "This application converts live commentary from sports events into sign language and Braille in real time, providing information to help visually and hearing-impaired individuals enjoy watching sports. Furthermore, it uses an emotion engine to recognize the user's emotions and highlights information corresponding to those emotions."

[0542] In this way, people with visual or hearing impairments can enjoy watching sports in real time, and information is emphasized according to the user's emotions, providing a more immersive viewing experience.

[0543] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[0544] Step 1:

[0545] The server uses the smartphone's microphone to acquire live audio data. The input is audio data from the smartphone's microphone, and the output is an audio data file. This audio data is used in subsequent processing steps.

[0546] Step 2:

[0547] The server converts the acquired audio data into text using the speech_recognition library. The input is the audio data acquired in step 1, and the output is text data. This conversion allows the audio information to be treated as text information.

[0548] Step 3:

[0549] The server uses a generative AI model to convert text data into sign language and Braille. The input is the text data obtained in step 2, and the output is sign language data and Braille data. This conversion uses a sign language conversion model and a Braille conversion model based on TensorFlow.

[0550] Step 4:

[0551] The server uses the emotion_recognition library to recognize the user's emotions in real time. The input is the text data obtained in step 2, and the output is the user's emotion data. This emotion data is used in subsequent information enhancement processing.

[0552] Step 5:

[0553] The server converts specific information into sign language or Braille, emphasizing it based on recognized emotion data. The input is the sign language and Braille data obtained in step 3, as well as the emotion data obtained in step 4, and the output is the emphasized sign language and Braille data. This process enables the provision of information tailored to the user's emotions.

[0554] Step 6:

[0555] The server outputs highlighted sign language and Braille data to the user. The input is the highlighted sign language and Braille data obtained in step 5, and the output is the sign language and Braille information displayed on the user's device. This allows visually and hearing impaired individuals to enjoy watching sports in real time.

[0556] (Example 3)

[0557] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0558] When users with visual or hearing impairments enjoy watching sports, there is a challenge in providing them with real-time information that responds to their emotions. In particular, the lack of information that takes users' emotions into consideration limits their experience, which is a problem.

[0559] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0560] In this invention, the server includes means for acquiring user emotion data, means for analyzing said emotion data, means for generating sign language or Braille data based on the analyzed emotion data, and means for outputting said sign language or Braille data. This makes it possible to provide real-time information in response to the user's emotions.

[0561] "User emotion data" refers to data that indicates the user's emotional state, obtained from the user's facial expressions, voice, and other sources.

[0562] "Means of acquisition" refers to devices and software used to collect user emotional data.

[0563] "Means of analysis" refers to devices or software used to analyze acquired emotional data and identify the user's emotional state.

[0564] "Generating means" refers to devices and software used to generate sign language or Braille data based on analyzed emotional data.

[0565] "Output methods" refer to devices and software used to provide users with generated sign language or Braille data.

[0566] "Sign language data" refers to data that contains information expressed in sign language.

[0567] "Braille data" refers to data that contains information expressed in Braille.

[0568] "Real-time" refers to data acquisition, analysis, generation, and output occurring without delay.

[0569] "Visually impaired or hearing impaired" refers to people who have a visual or hearing impairment.

[0570] Modes for carrying out the invention

[0571] This invention is a system that provides real-time information tailored to the user's emotions, enabling users with visual or hearing impairments to enjoy watching sports. Specific embodiments of this system are described below.

[0572] System Configuration

[0573] 1. Obtaining user sentiment data

[0574] The user inputs their facial expressions and voice through the device's camera and microphone. The device transmits this data to the server in real time. The hardware used includes a camera and microphone.

[0575] 2. Analysis of emotional data

[0576] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotions the user is feeling. For example, if the user has a sad expression, the emotion engine recognizes the emotion as "sadness." The software used includes the emotion engine.

[0577] 3. Generation of sign language and braille data

[0578] The server generates sign language and Braille data using a generative AI model based on emotional data obtained from the emotion engine. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if the user expresses sadness, the generative AI model will generate sign language and Braille data that emphasizes the sad scene.

[0579] 4. Data Output

[0580] The server transmits the generated sign language data to the terminal's display and the braille data to the braille display. The terminal displays the sign language on its screen and physically forms braille on the braille display. The user can receive information by looking at the display or touching the braille display. The hardware used includes a display and a braille display.

[0581] Specific example

[0582] As a concrete example, consider a scenario where a user is watching a sports event. If the user expresses sadness, the emotion engine recognizes this emotion, and the server uses a generative AI model based on this information to generate sign language or Braille data that emphasizes the sad scene. The generated data is then provided to the user through a display or Braille display.

[0583] Example of a prompt

[0584] "When a user expresses sadness while watching a sporting event, please recognize that emotion and generate sign language or Braille data that emphasizes the sad scene."

[0585] In this way, the server can provide information tailored to the user's emotions in real time, supporting visually and hearing impaired individuals in enjoying sports viewing. The flow of specific processing in Example 3 will be explained using Figure 21.

[0586] Step 1:

[0587] The user inputs their facial expressions and voice through the device's camera and microphone. The device transmits this data to the server in real time. The input consists of the user's facial expression data and voice data, while the output is emotion data sent to the server.

[0588] Step 2:

[0589] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotions the user is feeling. The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the emotion engine detects features such as "drooping eyebrows" and "downturned corners of the mouth" and determines that the user is expressing sadness.

[0590] Step 3:

[0591] The server generates sign language and Braille data using a generative AI model based on emotional data obtained from the emotion engine. The input is the analyzed emotional information, and the output is the generated sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if the user expresses sadness, the generative AI model will generate sign language and Braille data that emphasizes the sad scene.

[0592] Step 4:

[0593] The server transmits the generated sign language data to the terminal's display and the braille data to the braille display. The input is the generated sign language data and braille data, and the output is the sign language displayed on the screen and the braille formed on the braille display. The terminal displays the sign language on the screen and physically forms the braille on the braille display. The user can understand the sign language by looking at the display and read the braille by touching the braille display.

[0594] In this way, the server can provide information tailored to the user's emotions in real time, supporting people with visual or hearing impairments in enjoying watching sports.

[0595] (Application Example 3)

[0596] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0597] When users with visual or hearing impairments enjoy watching sports, there is a challenge in providing real-time information. Furthermore, the inability to provide information tailored to the user's emotions limits their experience.

[0598] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, means for generating sign language or Braille data according to the emotions, means for displaying the sign language data on a display, and means for outputting the Braille data to a Braille display. This makes it possible for users with visual or hearing impairments to enjoy watching sports in real time, and further enables the provision of information that is tailored to the user's emotions.

[0599] "Live broadcast audio data" refers to audio data that conveys the progress of sporting events and other live events in real time.

[0600] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[0601] Sign language is a visual, gesture-based language used by people with hearing impairments to communicate.

[0602] Braille is a writing system that uses raised dots to allow visually impaired people to read by touch.

[0603] A "display" is a device used to visually display information.

[0604] A "braille display" is a device that physically creates braille characters, allowing visually impaired people to read them by touch.

[0605] "Means of recognizing user emotions" refers to technologies that analyze and recognize emotions from the user's facial expressions, voice, etc.

[0606] "Means for generating sign language data" refers to technology that converts input information into a sign language format.

[0607] "Means for generating braille data" refers to technology that converts input information into a braille format.

[0608] "Means for displaying sign language data on a screen" refers to the technology for displaying generated sign language data on a screen.

[0609] "Means for outputting braille data to a braille display" refers to the technology for outputting generated braille data to a braille display.

[0610] The system for carrying out this invention has the following configuration.

[0611] System program

[0612] The system includes means for capturing live broadcast audio data, means for analyzing the audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting sign language or Braille, means for recognizing the user's emotions, means for generating sign language or Braille data according to the emotions, means for displaying the sign language data on a display, and means for outputting the Braille data to a Braille display.

[0613] Explanation of the process

[0614] Hardware to use

[0615] Smartphone camera

[0616] Smartphone display

[0617] Braille display

[0618] Software to use

[0619] OpenCV: Capturing and displaying camera footage

[0620] EmotionRecognizer: Emotion Recognition Model

[0621] SignLanguageGenerator: Sign language generation model

[0622] BrailleGenerator: Braille generation model

[0623] Data processing and calculations

[0624] The server captures the user's face with the smartphone's camera and inputs the video into the EmotionRecognizer to recognize the user's emotions. Based on the recognized emotions, the SignLanguageGenerator generates sign language data, and the BrailleGenerator generates braille data. The generated sign language data is displayed on the smartphone's screen, and the braille data is output to a braille display.

[0625] Specific example

[0626] For example, if a user expresses sadness while watching a sporting event, the system recognizes that emotion and highlights the sad scene, converting it into sign language or Braille. It can even provide a detailed explanation in sign language or Braille of a scene where a player from the losing team is crying.

[0627] Example of a prompt

[0628] Create a program that recognizes a user's emotions and generates sign language or Braille data corresponding to those emotions. Develop an application that, when a user expresses sadness, provides information in sign language or Braille that reflects that emotion.

[0629] The above describes the embodiments for carrying out this invention. This system enables users with visual or hearing impairments to enjoy watching sports in real time, and further enables the provision of information that responds to the user's emotions.

[0630] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[0631] Step 1:

[0632] The server captures the user's face using the smartphone's camera. The input is the camera feed, and the output is the captured image data. This image data is used as preprocessing for emotion recognition.

[0633] Step 2:

[0634] The server inputs the captured image data into the EmotionRecognizer to recognize the user's emotions. The input is image data, and the output is recognized emotion data. The EmotionRecognizer extracts facial features from the image data and classifies the emotions.

[0635] Step 3:

[0636] The server generates sign language data using SignLanguageGenerator based on recognized emotion data. The input is emotion data, and the output is sign language data. SignLanguageGenerator generates sign language actions corresponding to the emotion data.

[0637] Step 4:

[0638] The server generates Braille data using BrailleGenerator based on similarly recognized emotion data. The input is emotion data, and the output is Braille data. BrailleGenerator generates Braille patterns corresponding to the emotion data.

[0639] Step 5:

[0640] The server displays the generated sign language data on the smartphone's screen. The input is the sign language data, and the output is the sign language display on the screen. The display provides the user with a visual representation of the sign language actions.

[0641] Step 6:

[0642] The server outputs the generated braille data to a braille display. The input is braille data, and the output is a braille display on the braille display. The braille display physically forms braille patterns that the user can read by touch.

[0643] Step 7:

[0644] When a user expresses an emotion while watching a sporting event, the server recognizes that emotion in real time and generates and outputs appropriate sign language or Braille data. This allows users to receive information tailored to their emotions in real time.

[0645] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0646] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0647] Other examples of generative AI include Gemini® (registered trademark) (Internet search). <url: https: gemini.google.com ?hl="ja">) are some examples.

[0648] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0649] [Second Embodiment]

[0650] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0651] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0652] The data processing device 12 includes a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a “computer” related to the technology of this disclosure. The computer 22 includes a processor 28, RAM 30, and storage 32.

[0653] The processor 28, RAM 30, and storage 32 are connected to the bus 34. The database 24 and communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to the network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0654] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0655] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0656] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0657] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0658] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0659] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0660] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0661] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0662] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0663] "Example of form 1"

[0664] One embodiment of the present invention includes a microphone and audio input device for capturing audio data from live sports broadcasts. This audio input device can capture audio data from live broadcasts in real time. The captured audio data is transmitted to a generating AI.

[0665] "Example of form 2"

[0666] The generative AI analyzes captured audio data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the audio data. This generative AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[0667] "Example of form 3"

[0668] The generated sign language and braille data are provided to the user through an output device. Possible output devices include displays for showing sign language and braille displays for outputting braille. For example, a braille display physically forms braille based on the generated braille data, allowing visually impaired individuals to read it by touch.

[0669] The following describes the processing flow for each example of the form.

[0670] "Example of form 1"

[0671] Step 1: Install a microphone or audio input device to capture audio data from live sports broadcasts. This audio input device can capture audio data from live broadcasts in real time.

[0672] Step 2: The captured audio data is sent to the generating AI. This generating AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[0673] "Example of form 2"

[0674] Step 1: The generation AI analyzes the captured audio data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on the information obtained from the audio data.

[0675] Step 2: The generated sign language and braille data are provided to the user through an output device. Possible output devices include displays for showing sign language and braille displays for outputting braille.

[0676] "Example of form 3"

[0677] Step 1: As a concrete example, a braille display physically forms braille based on generated braille data, which can be read by visually impaired people through touch.

[0678] (Example 1)

[0679] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0680] There is a problem in that people with visual or hearing impairments have difficulty enjoying watching sports. In particular, they have difficulty understanding the audio information of live broadcasts in real time, which greatly diminishes their enjoyment of watching sports.

[0681] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for transmitting the live broadcast audio data to a generating AI model, and means for the generating AI model to analyze the audio data and convert it into text. This makes it possible for people with visual or hearing impairments to enjoy watching sports in real time.

[0682] "Live commentary audio data" refers to audio information provided in real time by commentators during events such as sports broadcasts.

[0683] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates a specific output.

[0684] "Means" refer to the methods or devices used to achieve a specific purpose.

[0685] "Converting to text" is the process of analyzing audio data and converting it into written information.

[0686] "Outputting" means providing processed data to the user in the form of display, audio, or other formats.

[0687] Modes for carrying out the invention

[0688] This invention is a system that captures audio data from live sports broadcasts in real time and transmits that data to a generating AI model. Specifically, it acquires audio data using a microphone or voice input device and transmits that data to a generating AI model to perform various data processing and calculations.

[0689] Hardware and software to use

[0690] Hardware: Microphone, audio input device

[0691] Software: Generative AI models (e.g., OpenAI's GPT-4)

[0692] System Overview

[0693] 1. Importing audio data:

[0694] Users provide live commentary on sports events.

[0695] Microphones and audio input devices connected to the terminal capture audio data from the live broadcast in real time.

[0696] 2. Sending audio data:

[0697] The captured audio data is sent from the terminal to the server.

[0698] The server sends the received audio data to the generating AI model.

[0699] 3. Data analysis and processing:

[0700] The generative AI model analyzes the received audio data and performs necessary data processing and calculations.

[0701] For example, it can perform tasks such as converting audio data to text, extracting important events, and generating real-time highlights.

[0702] 4. Output of results:

[0703] The server receives the results from the generated AI model and provides feedback to the user.

[0704] Specifically, the generated text is sent to the user's device, and the device displays that text.

[0705] Specific example

[0706] Specific example 1:

[0707] A user is providing live commentary on a soccer match.

[0708] The microphone picks up the voice saying "Goal!".

[0709] The device sends this audio data to the server.

[0710] The server sends audio data to the AI ​​model, which generates the text "Goal scored!".

[0711] Specific example 2:

[0712] A user is providing live commentary on a basketball game.

[0713] The microphone picks up the voice saying, "Three-point shot successful!"

[0714] The device sends this audio data to the server.

[0715] The server sends audio data to the AI ​​model, which generates the text "You made a three-point shot!".

[0716] Example of a prompt

[0717] Prompt message 1:

[0718] "Analyze this audio data and extract the important events. Audio data: 'Goal!'"

[0719] Prompt message 2:

[0720] "Please convert this audio data to text. Audio data: 'Three-point shot successful!'"

[0721] The above describes the embodiments for carrying out this invention.

[0722] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0723] Step 1:

[0724] The user provides live commentary on a sports event. A microphone and audio input device connected to the terminal capture the commentary audio data in real time. Specifically, when the user shouts "Goal!", the microphone captures the audio and converts it into digital data. The input is the user's voice, and the output is digital audio data.

[0725] Step 2:

[0726] The terminal sends the captured audio data to the server. Specifically, the terminal compresses the audio data and sends it to the server using the TCP / IP protocol. The input is digital audio data, and the output is the audio data sent to the server.

[0727] Step 3:

[0728] The server sends the received audio data to the generating AI model. Specifically, the server sends the audio data to the generating AI model's API and waits for the analysis results. The input is the audio data sent to the server, and the output is the audio data sent to the generating AI model.

[0729] Step 4:

[0730] The generative AI model analyzes the audio data and performs necessary data processing and calculations. Specifically, the generative AI model converts the audio data into text and generates the text "Goal scored!". The input is the audio data sent to the generative AI model, and the output is the generated text.

[0731] Step 5:

[0732] The server receives the results from the generated AI model and provides feedback to the user. Specifically, the server sends the generated text to the user's device, and the device displays that text. The input is the generated text, and the output is the text displayed on the user's device.

[0733] (Application Example 1)

[0734] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0735] In traditional sports viewing, it has been difficult for people with visual or hearing impairments to follow the progress of a match and follow important highlights in real time. Furthermore, general spectators have limited means of obtaining important match highlights and statistics in real time. This has resulted in a limitation in the enjoyment of watching sports.

[0736] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0737] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the transcribed live broadcast content, and means for outputting the generated highlights and statistical information. This makes it possible for people with visual or hearing impairments to obtain information in real time to enjoy watching sports, and also makes it possible for general spectators to obtain important highlights and statistical information of the match in real time.

[0738] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports and events.

[0739] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate a specific output.

[0740] Sign language is a language that uses hand and finger movements to communicate with people who are deaf or hard of hearing.

[0741] Braille is a tactile representation of writing that uses raised dots to allow visually impaired people to read text.

[0742] "Methods for converting to text" refer to technologies that analyze audio data and convert it into corresponding textual information.

[0743] A "highlight" refers to an important moment or event in a sport or event.

[0744] "Statistical information" refers to numerical data and analysis results related to matches and events.

[0745] "Means of output" refers to technologies such as display devices and audio devices that provide the generated information to the user.

[0746] A system for carrying out this invention includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the transcribed live broadcast content, and means for outputting the generated highlights and statistical information.

[0747] Hardware and software to be used

[0748] Hardware: Smartphone (with built-in microphone)

[0749] Software: Python, SpeechRecognition library (speech recognition), OpenAI API (generative AI)

[0750] Processing flow

[0751] 1. Importing audio data:

[0752] The smartphone's microphone is used to capture audio data from the live broadcast in real time. The user launches the smartphone application and begins voice input.

[0753] 2. Text conversion of audio data:

[0754] The captured audio data is converted to text using the SpeechRecognition library. The server sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data.

[0755] 3. Analysis using Generative AI:

[0756] The transcribed commentary is sent to a generating AI using the OpenAI API. The generating AI analyzes the input text data and generates important highlights and statistics.

[0757] 4. Information Output:

[0758] The generated highlights and statistics are displayed on the user's smartphone in real time. This allows users to stay informed about important moments and statistics from the match in real time.

[0759] Specific example

[0760] When a user is watching a soccer match, they launch a smartphone app to capture the audio from the live broadcast. The app converts the audio into text and displays important events such as "Goal!" and "Foul!" in real time. Furthermore, it automatically generates and displays match highlights and statistics (e.g., number of shots, ball possession).

[0761] Example of a prompt

[0762] Please generate the following sports commentary highlights:

[0763] "Team A scored a goal in the 30th minute of the first half. Team B's defense broke down. Team A continued their offensive in the second half as well."

[0764] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0765] Step 1:

[0766] The user launches a smartphone application to capture audio from a live broadcast. The user uses the smartphone's microphone to collect audio data in real time. The input is the audio data from the live broadcast, and the output is a stream of audio data.

[0767] Step 2:

[0768] The device converts the captured audio data into text using the SpeechRecognition library. The device then sends the audio data to the Google Speech Recognition API to retrieve the corresponding text data. The input is a stream of audio data, and the output is text data.

[0769] Step 3:

[0770] The server sends the transcribed commentary to the generating AI using the OpenAI API. The server analyzes the input text data and generates important highlights and statistics. The input is text data, and the output is the generated highlights and statistics.

[0771] Step 4:

[0772] The server sends the generated highlights and statistics to the user's smartphone in real time. The user can view this information on their smartphone screen. The input is the generated highlights and statistics, and the output is the information displayed on the smartphone screen.

[0773] Step 5:

[0774] Users can access important moments and statistics from a match in real time through a smartphone application. Specifically, users check the information on the screen and understand the progress of the match. The input is the information displayed on the smartphone screen, and the output is the user's understanding.

[0775] (Example 2)

[0776] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0777] There is a problem in that people with visual or hearing impairments have difficulty understanding auditory information in real time. In particular, the accuracy and speed of converting auditory information into sign language or Braille are insufficient, which can lead to delays in information transmission and misunderstandings. This creates a challenge in that it is difficult for people with visual or hearing impairments to receive information accurately and quickly.

[0778] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0779] In this invention, the server includes means for acquiring audio data, means for converting the audio data into text data using speech recognition software, and means for inputting the text data into a generation AI model to generate sign language data or Braille data. This makes it possible for visually and hearing impaired people to receive audio information in real time as sign language or Braille.

[0780] "Audio data" refers to data that records audio in digital format.

[0781] "Speech recognition software" is software that analyzes speech data and converts it into text data.

[0782] "Text data" refers to data that records character information in a digital format.

[0783] A "generative AI model" is an artificial intelligence model that is trained using deep learning technology to generate sign language data or Braille data based on text data.

[0784] "Sign language data" refers to data that records sign language movements in digital format.

[0785] "Braille data" refers to data that records Braille information in digital format.

[0786] "Sign language animation software" is software that visually displays sign language actions based on sign language data.

[0787] A "braille display" is a device that physically displays braille based on braille data.

[0788] This invention is a system that enables people with visual or hearing impairments to receive audio information in real time as sign language or Braille. Specific embodiments of this system are described below.

[0789] System Configuration

[0790] Server Configuration

[0791] The server includes means for acquiring audio data, means for converting audio data into text data using speech recognition software, and means for converting text data into sign language data or Braille data using a generative AI model. For speech recognition software, a common speech recognition API (e.g., Google Cloud Speech-to-Text API) is used. The generative AI model is a model trained using deep learning techniques (e.g., a model using TensorFlow or PyTorch).

[0792] Device configuration

[0793] The terminal includes means for visually displaying sign language data transmitted from the server using sign language animation software (e.g., SignAll), and means for outputting braille data to a braille display (e.g., Focus Blue). The terminal also includes a microphone for the user to input voice data.

[0794] User actions

[0795] The user inputs voice data by speaking into the device's microphone. For example, they might say "hello." The device sends this voice data to a server. The server converts the voice data into text data and uses a generative AI model to generate sign language data or Braille data. The generated data is sent to the device and displayed as a sign language animation on the screen or output to a Braille display.

[0796] Specific example

[0797] Example 1: Conversion to sign language

[0798] When a user says "Good morning," the device sends the audio data to a server. The server converts the audio data into text and inputs the text information "Good morning" into a generative AI model. The generative AI model generates sign language data corresponding to "Good morning" and creates a sign language animation using sign language animation software. The device displays this sign language animation on its screen, allowing the user to visually confirm the sign language for "Good morning."

[0799] Example 2: Conversion to Braille

[0800] When the user says "thank you," the device sends the audio data to a server. The server converts the audio data into text and inputs the text information "thank you" into a generating AI model. The generating AI model generates braille data corresponding to "thank you" and sends it to a braille display. The braille display connected to the device displays "thank you" in braille, and the user can read the braille by touch.

[0801] Example of a prompt

[0802] "Please explain how the generative AI model translates a user's 'Good morning' into sign language or Braille."

[0803] By inputting this prompt into the generating AI model, a detailed explanation of the specific transformation process can be obtained.

[0804] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0805] Step 1:

[0806] The user inputs voice data.

[0807] The user speaks into the device's microphone. For example, they might say "Hello." The device captures this voice as digital audio data. The input is the user's voice, and the output is digital audio data.

[0808] Step 2:

[0809] The device sends voice data to the server.

[0810] The device transmits the captured audio data to the server via the internet. During this process, the data is encoded in an appropriate format (e.g., WAV or MP3). The input is digital audio data, and the output is the audio data transmitted to the server.

[0811] Step 3:

[0812] The server converts the audio data into text data.

[0813] The server converts the received audio data into text data using speech recognition software (e.g., a common speech recognition API). For example, the audio data "Hello" is converted into the text data "Hello". The input is audio data, and the output is text data.

[0814] Step 4:

[0815] The server generates text data and inputs it into the AI ​​model.

[0816] The server inputs the converted text data into a generative AI model. This generative AI model is trained using deep learning techniques. The input is text data, and the output is sign language data or Braille data.

[0817] Step 5:

[0818] The generative AI model generates sign language data and Braille data.

[0819] The generative AI model generates sign language data and Braille data based on input text data. For example, the text data "Hello" is converted into sign language animation data and Braille data. The input is text data, and the output is sign language data and Braille data.

[0820] Step 6:

[0821] The server sends the generated data to the terminal.

[0822] The server sends the generated sign language or braille data to the terminal. The data is encoded in an appropriate format (e.g., JSON or XML). The input is the sign language or braille data, and the output is the data sent to the terminal.

[0823] Step 7:

[0824] The terminal displays sign language animations and outputs Braille data to a Braille display.

[0825] The terminal displays received sign language data on its screen using sign language animation software (e.g., sign language animation software). Braille data is output to a braille display connected to the terminal (e.g., a braille display). The user can visually confirm the sign language animation displayed on the screen and tactilely read the braille displayed on the braille display. Input is sign language data and braille data, and output is sign language animation and braille.

[0826] (Application Example 2)

[0827] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0828] There is a problem in that visually impaired and hearing-impaired individuals have difficulty understanding voice instructions and warnings in real time during security work. This can prevent them from performing their duties efficiently, potentially leading to a decline in the quality of security.

[0829] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the audio data into text, means for converting the text into sign language and Braille, and means for providing the sign language and Braille to visually impaired and hearing impaired people. This makes it possible for visually impaired and hearing impaired people to understand voice instructions and warnings in real time and perform security work efficiently.

[0830] "Live broadcast audio data" refers to audio information of events and situations that are taking place in real time.

[0831] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[0832] "Sign language" refers to a means of conveying linguistic information using visual gestures and movements.

[0833] Braille refers to a writing system that uses raised dots to allow visually impaired people to read text information using their sense of touch.

[0834] "Audio data" refers to information recorded in digital format using audio.

[0835] "Text" refers to written information.

[0836] "Visually impaired" refers to people who have a visual impairment.

[0837] "People with hearing impairments" refers to individuals who have a hearing loss.

[0838] "Security work" refers to work performed to protect the safety of facilities and people.

[0839] "Real-time" refers to processing or communication occurring instantly without delay.

[0840] The system for carrying out this invention converts audio data into sign language or Braille in real time and provides it to visually impaired and hearing impaired individuals. Specific embodiments of the system are described below.

[0841] System Configuration

[0842] hardware

[0843] Smartphone: Equipped with a microphone for acquiring audio data and a display for showing the conversion results.

[0844] Server: Provides computing resources for analyzing and converting audio data.

[0845] software

[0846] Speech recognition library: Use the speech_recognition library to convert speech data to text.

[0847] Sign language conversion library: Use the hand_gesture_recognition library to convert text into sign language.

[0848] Braille conversion library: Use the braille_converter library to convert text to Braille.

[0849] Generative AI Model: Uses a deep learning model to analyze audio data and convert it into sign language or Braille.

[0850] System operation

[0851] Acquisition of audio data

[0852] The user inputs voice data using their smartphone's microphone. For example, a security guard might input "There is a suspicious person" by voice.

[0853] Analysis of audio data

[0854] The server converts the acquired audio data into text using the speech_recognition library. This text forms the basis for subsequent sign language and Braille conversion.

[0855] Conversion to sign language and Braille

[0856] The server uses a generative AI model to convert text into sign language and Braille. The `hand_gesture_recognition` library is used for sign language conversion, and the `braille_converter` library is used for Braille conversion.

[0857] Providing results

[0858] The converted sign language and Braille are displayed on the smartphone screen. This allows visually impaired and hearing-impaired individuals to understand voice instructions and warnings in real time.

[0859] Specific example

[0860] For example, if a security guard voice-inputs "There is a suspicious person," the system converts this voice into text, and then into sign language and Braille. The results are displayed on a smartphone screen, ensuring that the information is also conveyed to security guards who are visually impaired or hearing impaired.

[0861] Example of a prompt

[0862] Please enter the audio data. Example: "There is a suspicious person."

[0863] In this way, visually impaired and hearing-impaired individuals can understand voice instructions and warnings in real time, enabling them to perform security tasks efficiently.

[0864] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0865] Step 1:

[0866] The user inputs voice data using the smartphone's microphone. The input voice data is temporarily stored in the smartphone's internal memory.

[0867] Step 2:

[0868] The smartphone sends the stored audio data to the server. The server receives the audio data and uses the speech_recognition library to convert it to text. The input is audio data, and the output is text data.

[0869] Step 3:

[0870] The server analyzes text data using a generative AI model. This analysis converts the text data into sign language and Braille. The input is text data, and the output is sign language data and Braille data.

[0871] Step 4:

[0872] The server converts sign language data into sign language using the hand_gesture_recognition library. The input is text data, and the output is sign language data.

[0873] Step 5:

[0874] The server converts braille data into braille using the braille_converter library. The input is text data, and the output is braille data.

[0875] Step 6:

[0876] The server sends the converted sign language and Braille data to the smartphone. The smartphone displays the received sign language and Braille data on its screen. This allows visually impaired and hearing-impaired individuals to understand voice instructions and warnings in real time.

[0877] (Example 3)

[0878] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0879] The aim is to solve the problem that users with visual or hearing impairments have difficulty converting text data into sign language or Braille and accessing information in real time. In particular, there is a need for a system that can quickly and accurately convert text data entered by users into sign language or Braille and provide it through an appropriate output device.

[0880] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0881] In this invention, the server includes means for a user to input text data, means for the server to receive the input data and send prompt sentences to a generating AI model, means for the generating AI model to generate sign language or Braille data, means for the server to transmit the generated data to an output device, and means for the terminal to provide sign language or Braille to the user through the output device. This makes it possible for users with visual or hearing impairments to quickly and accurately convert the input text data into sign language or Braille and use the information in real time.

[0882] A "user" refers to a person who uses the system to input text data and receives the information converted into sign language or Braille.

[0883] A "server" refers to a computer system that receives input data from users, sends prompt messages to the AI ​​generation model, and transmits the generated sign language or Braille data to an output device.

[0884] A "generative AI model" refers to an artificial intelligence model that generates sign language or Braille data based on text data from users.

[0885] A "prompt sentence" refers to a sentence used to instruct a generative AI model to convert text data into sign language or Braille.

[0886] An "output device" is a device that provides generated sign language or Braille data to the user, and includes sign language displays and Braille displays.

[0887] A "sign language display" refers to a display device used to visually display generated sign language data.

[0888] A "braille display" refers to a device that physically forms generated braille data, allowing visually impaired individuals to read it by touch.

[0889] "Text data" refers to the character information that a user enters into the system.

[0890] "Real-time" refers to the process where text data entered by a user is instantly converted into sign language or Braille and provided almost simultaneously through an output device.

[0891] Modes for carrying out the invention

[0892] This invention is a system that enables users with visual or hearing impairments to convert text data into sign language or Braille and access information in real time. Specific embodiments of this system are described below.

[0893] System Configuration

[0894] This system consists of the following main components:

[0895] 1. User terminal: A device used by a user to input text data. Examples include personal computers, smartphones, and tablets.

[0896] 2. Server: This server receives input data from the user, sends prompt messages to the generation AI model, and sends the generated sign language or Braille data to the output device. A high-performance server (e.g., a server equipped with an NVIDIA GPU) should be used.

[0897] 3. Generative AI Model: An artificial intelligence model for generating sign language or Braille data based on text data from users. It uses deep learning frameworks such as TensorFlow or PyTorch.

[0898] 4. Output device: A device for providing the generated sign language or Braille data to the user. This includes sign language displays and Braille displays.

[0899] System operation

[0900] 1. The user enters text data.

[0901] Users input text data through the system interface. For example, they might type "Hello" using a web browser or a dedicated application.

[0902] 2. The server receives the input data and sends a prompt message to the generating AI model.

[0903] The server receives text data sent by the user. Based on the received data, it creates and sends a prompt to the generative AI model. For example, it might send the prompt "Please convert the following text into sign language: Hello" to the generative AI model.

[0904] 3. The generative AI model generates data for sign language and Braille.

[0905] The generative AI model analyzes prompt messages received from the server and generates corresponding sign language or Braille data. For example, it can generate sign language animation data or Braille sequence data.

[0906] 4. The server sends the generated data to the output device.

[0907] The server transmits the sign language and Braille data received from the generated AI model to the output devices. For example, it transmits sign language animation data to a sign language display and Braille data to a Braille display.

[0908] 5. The terminal provides sign language or Braille to the user through an output device.

[0909] The terminal receives data transmitted from the server and provides it to the user through an output device. The sign language display shows sign language animations, and the braille display physically forms braille characters. Users can visually confirm the sign language display and tactilely read the braille display.

[0910] Specific example

[0911] For example, the specific actions taken when a user enters the text "thank you" are as follows:

[0912] 1. User: Type "Thank you" into the input field of the web browser and click the submit button.

[0913] 2. Server: Receives the text data "thank you" from the user and sends the prompt message "Please convert the following text into sign language: thank you" to the AI ​​model.

[0914] 3. Generative AI Model: Analyzes the prompt sentence and generates sign language animation data corresponding to "thank you".

[0915] 4. Server: Sends the generated sign language animation data to the sign language display.

[0916] 5. Terminal: The sign language display shows sign language animations for the user to visually confirm.

[0917] In this way, users can easily convert text into sign language or Braille and use it through an output device. The flow of the specific processing in Example 3 will be explained with reference to Figure 15.

[0918] Step 1:

[0919] The user enters text data.

[0920] Users input text data using a web browser or a dedicated application. For example, they might type "Hello" and click the submit button. The input data is then sent from the user's terminal to the server.

[0921] Step 2:

[0922] The server receives the input data and sends prompt messages to the generating AI model.

[0923] The server receives text data sent by the user. Based on the received data, it creates and sends a prompt to the generative AI model. For example, it might send the prompt "Please convert the following text into sign language: Hello" to the generative AI model. The input is the user's text data, and the output is the prompt to the generative AI model.

[0924] Step 3:

[0925] The generative AI model generates data for sign language and Braille.

[0926] The generative AI model analyzes prompt text received from the server and generates corresponding sign language or Braille data. For example, it can generate sign language animation data or Braille sequence data. The input is prompt text, and the output is sign language or Braille data.

[0927] Step 4:

[0928] The server sends the generated data to the output device.

[0929] The server transmits the sign language and Braille data received from the generating AI model to the output device. For example, it transmits sign language animation data to a sign language display and Braille data to a Braille display. The input is the generated sign language and Braille data, and the output is the transmission of data to the output device.

[0930] Step 5:

[0931] The terminal provides sign language and Braille to the user through an output device.

[0932] The terminal receives data transmitted from the server and provides it to the user through an output device. The sign language display shows sign language animations, and the braille display physically forms braille characters. The user can visually confirm the sign language display and tactilely read the braille display. The input is sign language or braille data from the server, and the output is the provision of sign language or braille to the user.

[0933] (Application Example 3)

[0934] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 as a "terminal".

[0935] There is a problem in that people with visual or hearing impairments have difficulty receiving real-time information when watching sports or obtaining security information. Furthermore, there is a need for rapid information transmission in sign language and Braille during emergencies, but the current system is insufficient to meet this need.

[0936] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for acquiring video from a security camera, analyzing it with a sign language recognition model and recognizing sign language, means for displaying the recognized sign language on the video, and means for outputting security information to a Braille display. This makes it possible for people with visual or hearing impairments to watch sports and receive security information in real time.

[0937] "Live broadcast audio data" refers to data that conveys the progress of sporting events and other live events through audio.

[0938] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[0939] Sign language is a visual, gesture-based language used by people with hearing impairments to communicate.

[0940] Braille is a writing system that uses raised dots to allow visually impaired people to read by touch.

[0941] A "sign language recognition model" is a machine learning model that analyzes sign language movements from camera footage and other sources and converts them into the corresponding language.

[0942] A "security camera" is a camera device used to monitor a specific area and acquire video data.

[0943] A "braille display" is a device that displays braille using electronically controlled raised dots.

[0944] "Security information" refers to information regarding anomalies or emergencies within the monitored area.

[0945] The system for carrying out this invention is configured as follows: The server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for acquiring video from a security camera, analyzing it with a sign language recognition model and recognizing sign language, means for displaying the recognized sign language on the video, and means for outputting security information to a Braille display.

[0946] Hardware and software to be used

[0947] Hardware:

[0948] Smartphone camera

[0949] Braille display (e.g., BrailleDisplay)

[0950] Security camera

[0951] software:

[0952] OpenCV: Acquiring and displaying camera images

[0953] HandSignRecognition: Sign Language Recognition Model

[0954] Braille Display: Braille display control library

[0955] Data processing and data calculation

[0956] 1. Importing live broadcast audio data:

[0957] The server receives audio data from live broadcasts that provide updates on the progress of sporting events and other live events.

[0958] 2. Analysis using Generative AI:

[0959] The server uses AI to analyze the captured live broadcast audio data and converts it into sign language or Braille in real time.

[0960] 3. Output of sign language and Braille:

[0961] The server outputs the generated sign language or Braille through a display or Braille display.

[0962] 4. Capture video from security cameras:

[0963] The server captures video from security cameras and analyzes it using a sign language recognition model.

[0964] 5. Recognition and display of sign language:

[0965] The server uses a sign language recognition model to recognize sign language in the video and displays the recognized sign language on the video.

[0966] 6. Outputting security information to a Braille display:

[0967] The server outputs security information to a braille display.

[0968] Specific example

[0969] Watching sports:

[0970] Users with visual or hearing impairments can receive information via displays or braille displays by having live audio data from sports events converted into sign language or braille in real time.

[0971] Obtaining security information:

[0972] Security cameras capture video footage, a sign language recognition model recognizes the sign language, and the recognized sign language is displayed on the video. Additionally, security information is output to a Braille display, allowing visually impaired individuals to receive information through touch.

[0973] Example of a prompt

[0974] "Create a program that analyzes security camera footage, displays emergency information in sign language, and outputs security information to a Braille display."

[0975] The above describes the embodiments for carrying out this invention.

[0976] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[0977] Step 1:

[0978] The server receives audio data from live broadcasts of sporting events and other live events, providing commentary on the progress of those events. The input is the live broadcast audio data, and the output is the captured audio data. This audio data is used for subsequent analysis.

[0979] Step 2:

[0980] The server analyzes the incoming live broadcast audio data using a generating AI and converts it into sign language and Braille in real time. The input is live broadcast audio data, and the output is sign language data and Braille data. The generating AI analyzes the audio data and generates the corresponding sign language and Braille.

[0981] Step 3:

[0982] The server outputs the generated sign language and Braille through a display or Braille display. Input is sign language data and Braille data, and output is sign language display on the display and Braille display on the Braille display. This allows users with visual or hearing impairments to receive information.

[0983] Step 4:

[0984] The server captures video from security cameras and analyzes it using a sign language recognition model. The input is video data from the security cameras, and the output is recognized sign language data. The sign language recognition model analyzes the video data and recognizes sign language movements.

[0985] Step 5:

[0986] The server displays the recognized sign language on the video. The input is the recognized sign language data, and the output is the sign language displayed on the video. This allows the user to understand security camera footage in sign language.

[0987] Step 6:

[0988] The server outputs security information to a braille display. The input is security information, and the output is a braille display on the braille display. This allows the user to receive security information through touch.

[0989] The above are the specific processing steps for carrying out this invention.

[0990] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0991] "Example of form 1"

[0992] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes the user's emotions and generates information to help visually or hearing-impaired individuals enjoy watching sports more. Specifically, it estimates emotions from the user's facial expressions, tone of voice, and word choice, and analyzes live broadcast audio data according to those emotions. For example, if the user is showing emotion of joy, the system recognizes that emotion and highlights more exciting scenes, converting them into sign language or Braille.

[0993] "Example of form 2"

[0994] Furthermore, the emotion engine recognizes emotions in real time and generates information accordingly. This makes it possible to provide information that allows visually and hearing-impaired individuals to enjoy watching sports in accordance with the emotions the user is feeling in real time. Specifically, if a user expresses surprise, the system recognizes that emotion, emphasizes the surprising scene, and converts it into sign language or Braille.

[0995] "Example of form 3"

[0996] Furthermore, the emotion engine generates information in response to the user's emotions and outputs that information in real time. This makes it possible to instantly provide information that allows visually and hearing-impaired individuals to enjoy watching sports, tailored to the emotions the user is feeling. Specifically, if a user expresses sadness, the system recognizes that emotion, emphasizes the sad scene, and converts it into sign language or Braille.

[0997] The following describes the processing flow for each example of the form.

[0998] "Example of form 1"

[0999] Step 1: Estimate the user's emotions from their facial expressions, tone of voice, and word choice.

[1000] Step 2: Analyze the live commentary audio data based on the estimated emotions.

[1001] Step 3: Based on the analysis results, highlight the most exciting scenes and convert them into sign language or Braille.

[1002] "Example of form 2"

[1003] Step 1: Recognize the user's emotions in real time.

[1004] Step 2: Analyze the live commentary audio data based on the recognized emotions.

[1005] Step 3: Based on the analysis results, highlight the surprising scenes and convert them into sign language or Braille.

[1006] "Example of form 3"

[1007] Step 1: Recognize the user's emotions in real time and generate information in response to those emotions.

[1008] Step 2: Output the generated information in real time.

[1009] Step 3: Based on the output information, emphasize the sad scenes and convert them into sign language or Braille.

[1010] (Example 1)

[1011] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1012] When users with visual or hearing impairments enjoy watching sports, they face the challenge of understanding the audio information in live broadcasts in real time. Furthermore, the inability to provide information that aligns with the user's emotions means that the enjoyment of watching the game is not fully conveyed.

[1013] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generation AI model and recognizing the user's emotions, and means for converting the live broadcast audio data into sign language or Braille in real time according to the user's emotions. This makes it possible to provide real-time information for users with visual or hearing impairments to enjoy watching sports.

[1014] "Live commentary audio data" refers to data that transmits the progress of sporting events and other live events in real time via audio.

[1015] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to analyze data and generate information tailored to a specific purpose.

[1016] "Means of recognizing user emotions" refer to technologies and devices that analyze a user's facial expressions, tone of voice, word choice, etc., to estimate their emotional state.

[1017] "Means of converting to sign language or Braille" refer to software and hardware for converting audio data or text data into sign language videos or Braille information.

[1018] "Means of outputting sign language or Braille" refer to output devices such as displays and Braille displays that provide users with generated sign language videos and Braille information.

[1019] An "exciting scene" refers to a particularly important moment or play in a sporting event that evokes excitement or emotion.

[1020] This invention is a system for users with visual or hearing impairments to enjoy watching sports. This system captures live broadcast audio data in real time, recognizes the user's emotions, and provides information corresponding to those emotions in sign language or Braille.

[1021] Hardware and software to use

[1022] hardware

[1023] Microphones: Installed in stadiums and event venues to capture audio data of live broadcasts in real time.

[1024] Camera: Used to capture the user's facial expressions.

[1025] Terminal: A device used to process audio and video data and send it to a server.

[1026] Braille display: An output device for providing Braille information to users.

[1027] software

[1028] Generative AI model: An algorithm that analyzes voice data and recognizes the user's emotions.

[1029] Emotion engine: Software used to estimate emotions from a user's facial expressions, tone of voice, word choice, etc.

[1030] Sign language translation software: Software used to convert audio or text data into sign language videos.

[1031] Braille conversion software: Software used to convert audio data and text data into Braille information.

[1032] Specific operation of the system

[1033] Importing audio data

[1034] The device uses high-sensitivity microphones installed in the stadium to capture audio data from the live broadcast in real time. For example, it can capture the audio at the moment the commentator shouts "Goal!".

[1035] Sending audio data

[1036] The device transmits the captured audio data to a generative AI model in the cloud via the internet. Specifically, the audio data is compressed and sent to the server using a secure communication protocol.

[1037] User emotion recognition

[1038] The server monitors whether the user is connected to the system via camera and microphone. The emotion engine analyzes the user's facial expressions (e.g., smiles or expressions of surprise) and voice tone (e.g., excited voice) in real time to estimate the user's emotions.

[1039] Emotion-based data analysis

[1040] If the server estimates that the user's emotion is one of joy, it will highlight exciting scenes from the live commentary audio data. For example, it will identify and extract moments of goals or important plays.

[1041] Information generation and provision

[1042] The server converts the extracted exciting scenes into sign language and Braille. Specifically, it uses sign language translation software to generate sign language videos and Braille conversion software to display them on a Braille display. The generated information is sent to the user's terminal and displayed in real time.

[1043] Specific examples and prompt statements

[1044] Specific example

[1045] Let's say a user is watching a soccer match. If the user shows an expression of joy during a goal, the system recognizes that emotion and emphasizes the commentary about the goal, converting it into sign language or Braille.

[1046] Example of a prompt

[1047] "Please capture the live commentary audio of a goal scored during a soccer match in real time, recognize the user's emotions of joy, and convert that scene into sign language or Braille."

[1048] In this way, the system provides information tailored to the user's emotions, enabling people with visual or hearing impairments to enjoy watching sports more.

[1049] The flow of the specific processing in Example 1 will be explained using Figure 17.

[1050] Step 1:

[1051] The device uses high-sensitivity microphones installed in the stadium to capture audio data from the live broadcast in real time. The input is the audio data from the live broadcast, and the output is the captured audio data. Specifically, it uses a noise-canceling function to remove background noise.

[1052] Step 2:

[1053] The device transmits the captured audio data to a generative AI model in the cloud via the internet. The input is the captured audio data, and the output is the audio data sent to the generative AI model. Specifically, the audio data is compressed and sent to the server using a secure communication protocol.

[1054] Step 3:

[1055] The server monitors the user's connection to the system via camera and microphone. Inputs are the user's facial expression data and voice tone data, and outputs are the data sent to the emotion engine. Specifically, the camera captures the user's facial expressions, and the microphone collects the user's voice tone.

[1056] Step 4:

[1057] The server uses an emotion engine to estimate the user's emotions from their facial expressions, tone of voice, and word choice. The input is the user's facial expression data and tone of voice data, and the output is the estimated emotion data. Specifically, the emotion engine analyzes the data in real time to estimate the user's emotions.

[1058] Step 5:

[1059] If the server estimates the user's emotion to be joy, it highlights exciting scenes from the live commentary audio data. The input is the estimated emotion data and the live commentary audio data, and the output is the data of the highlighted exciting scenes. Specifically, it uses an algorithm to detect certain keywords (e.g., "goal," "great play") contained in the live commentary audio data.

[1060] Step 6:

[1061] The server converts the extracted exciting scenes into sign language or Braille. The input is data of the highlighted exciting scenes, and the output is sign language video or Braille information. Specifically, it uses sign language translation software and Braille conversion software to convert the data.

[1062] Step 7:

[1063] The server transmits the generated sign language video and Braille information to the user's terminal. The input is the sign language video and Braille information, and the output is the sign language video and Braille information displayed on the user's terminal. Specifically, a low-latency communication protocol is used to ensure real-time operation.

[1064] In this way, the system provides information tailored to the user's emotions, enabling people with visual or hearing impairments to enjoy watching sports more.

[1065] (Application Example 1)

[1066] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1067] When users with visual or hearing impairments enjoy watching sports, simply converting the audio data of live broadcasts into sign language or Braille in real time presents a challenge in providing information that aligns with the user's emotions. Furthermore, the lack of emphasis on exciting scenes that take the user's emotions into consideration makes it difficult for visually and hearing-impaired individuals to fully enjoy watching sports.

[1068] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1069] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, and means for highlighting exciting scenes according to those emotions. This makes it possible to provide information that responds to the user's emotions so that visually and hearing impaired people can enjoy watching sports more.

[1070] "Live broadcast audio data" refers to audio information that conveys the progress of sporting events and other live events in real time.

[1071] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate a specific output.

[1072] Sign language is a means of conveying linguistic information using visual gestures and movements.

[1073] Braille is a pattern of dots raised on paper or other surfaces so that visually impaired people can read it using their sense of touch.

[1074] "Means of recognizing user emotions" refers to technologies that estimate emotions from a user's facial expressions, tone of voice, word choice, etc.

[1075] "Methods for emphasizing exciting scenes" refers to techniques for selecting particularly exciting moments in sports viewing and emphasizing those moments to people with visual or hearing impairments.

[1076] The system for carrying out this invention is configured as follows: First, the server is equipped with a microphone and an audio input device for capturing live broadcast audio data. This audio input device can capture live broadcast audio data of sports events in real time.

[1077] Next, the captured audio data is sent to a generating AI. The generating AI analyzes the audio data and converts it into sign language or Braille in real time. This converted sign language or Braille data is output in a format that can be understood by people with visual or hearing impairments.

[1078] Furthermore, the system is equipped with an emotion recognition engine to recognize the user's emotions. This engine estimates emotions from the user's facial expressions, tone of voice, and word choice. The emotion recognition engine acquires and analyzes the user's emotional data using a camera and microphone.

[1079] When a user's emotions are recognized, the system highlights exciting scenes accordingly. For example, if a user expresses joy, the system recognizes this emotion and selects and highlights particularly exciting scenes. This highlighted information is then translated into sign language or Braille and provided to people with visual or hearing impairments.

[1080] The specific hardware used includes a microphone, camera, and server. The software used includes Python, the speech_recognition library, the transformers library, and the opencv library. Combining this hardware and software enables the capture of speech data, emotion recognition, conversion to sign language or Braille, and highlighting of exciting scenes.

[1081] For example, if a user expresses feelings of joy while watching a sporting event, the system recognizes those feelings and translates them into sign language or Braille, highlighting particularly exciting scenes. This information is provided to the user in real time.

[1082] Examples of prompt statements to input into a generative AI model include the following:

[1083] If a user is expressing feelings of joy, how can I highlight the exciting scenes and translate them into sign language or Braille?

[1084] This prompt can be used to query the generative AI model for specific sign language or Braille conversion methods.

[1085] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[1086] Step 1:

[1087] The server uses a microphone to capture live commentary audio data in real time.

[1088] Input: Live broadcast audio data

[1089] Output: Captured audio data

[1090] Specific operation: The server captures the audio signal from the microphone as digital data and performs preprocessing for speech recognition.

[1091] Step 2:

[1092] The server sends the captured audio data to the AI ​​for analysis.

[1093] Input: Captured audio data

[1094] Output: Analyzed audio data

[1095] Specific operation: The server inputs the audio data into a generation AI model, which then performs speech recognition and natural language processing. This converts the audio data into text data.

[1096] Step 3:

[1097] The server converts the analyzed audio data into sign language or Braille in real time.

[1098] Input: Analyzed audio data

[1099] Output: Sign language data, Braille data

[1100] Specific operation: The server executes an algorithm that converts text data into sign language or Braille using a generative AI model. The sign language data is output in video format, and the Braille data is output in a data format for tactile displays.

[1101] Step 4:

[1102] The server uses a camera to capture the user's facial expressions and sends them to the emotion recognition engine.

[1103] Input: User's facial expression data

[1104] Output: Recognized emotion data

[1105] Specific operation: The server acquires video data from the camera and inputs it into the emotion recognition engine. The emotion recognition engine performs facial expression analysis and estimates the user's emotions.

[1106] Step 5:

[1107] The server highlights exciting scenes based on recognized emotion data.

[1108] Input: Recognized emotion data, sign language data, braille data

[1109] Output: Highlighted sign language data, Highlighted braille data

[1110] Specific operation: The server analyzes emotional data and, if the user is expressing feelings of joy, selects and highlights particularly exciting scenes. This adjusts the sign language and braille data according to the user's emotions.

[1111] Step 6:

[1112] The server provides users with highlighted sign language and Braille data.

[1113] Input: Highlighted sign language data, Highlighted Braille data

[1114] Output: Sign language data and Braille data provided to the user.

[1115] Specific operation: The server sends highlighted sign language data to the user in video format and braille data in a data format for tactile displays. The user receives this data in real time and can enjoy watching sports.

[1116] (Example 2)

[1117] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1118] When users with visual or hearing impairments enjoy watching sports, they face the challenge of understanding the audio information in live broadcasts in real time. Furthermore, because information is not provided in a way that responds to the user's emotions, it is difficult to share the sense of presence and emotion.

[1119] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1120] In this invention, the server includes means for acquiring live broadcast audio data, means for preprocessing the live broadcast audio data, means for analyzing the preprocessed audio data using a generation AI model and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions from the audio data, and means for generating information based on the recognized emotions. This makes it possible for users with visual or hearing impairments to enjoy watching sports in real time.

[1121] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports events and other events.

[1122] "Preprocessing" refers to processes such as noise reduction and normalization performed to convert audio data into a format that is easier to analyze.

[1123] A "generative AI model" is an artificial intelligence model trained using deep learning technology to analyze speech data and convert it into sign language or Braille.

[1124] Sign language is a visual gesture language used by visually impaired people to understand auditory information.

[1125] Braille is a writing system that uses raised dots to allow visually impaired people to understand written information through touch.

[1126] "Emotion recognition" is an analytical technique used to identify a user's emotions from voice data.

[1127] "Information generation" is the process of creating information to provide to the user based on analyzed voice data and recognized emotions.

[1128] "Output means" refers to devices or interfaces for providing generated sign language or Braille information to the user.

[1129] This invention is a system for users with visual or hearing impairments to enjoy watching sports in real time. Specific embodiments of this system are described below.

[1130] System Configuration

[1131] hardware

[1132] The server uses a microphone (e.g., a standard microphone) to capture audio data. It also has a high-performance processor and memory for processing and analyzing the audio data. Furthermore, it is equipped with a display and a braille display for outputting sign language and braille information.

[1133] software

[1134] The server uses Librosa, a Python library, to preprocess audio data. The generative AI model is built using TensorFlow and PyTorch, and analyzes audio data to convert it into sign language or Braille. OpenCV and the Emotion API are used for emotion recognition.

[1135] System operation

[1136] The server captures live broadcast audio data in real time via a microphone. The captured audio data is preprocessed using Librosa, including noise reduction and normalization. The preprocessed audio data is then input into a generative AI model and converted into sign language or Braille.

[1137] Simultaneously, the server uses an emotion engine to recognize the user's emotions in real time from the audio data. Based on the recognized emotions, information tailored to the user's feelings is generated. For example, if the user expresses surprise, the system recognizes that emotion and highlights the surprising scene, converting it into sign language or Braille.

[1138] The generated sign language and Braille information is provided to the user via the terminal. In the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1139] Specific example

[1140] If a user exclaims "Amazing!" while watching a sporting event, the server captures the audio and inputs it into a generative AI model. The generative AI model analyzes the audio and generates data that converts the exclamation "Amazing!" into sign language or Braille. Simultaneously, an emotion engine recognizes the user's emotion of surprise and generates information that emphasizes the surprising scene according to that emotion.

[1141] Example of a prompt

[1142] "Please describe a system that, when a user exclaims in surprise while watching a sporting event, converts that sound into sign language or Braille in real time, highlighting the surprising moment and providing information to people with visual or hearing impairments."

[1143] This system allows users with visual or hearing impairments to enjoy watching sports in real time.

[1144] The flow of the specific processing in Example 2 will be explained using Figure 19.

[1145] Step 1:

[1146] The server captures live broadcast audio data in real time via a microphone. The input is analog audio data, which is converted to a digital format and stored in temporary storage. Specifically, the audio input device captures the audio and performs digital signal processing.

[1147] Step 2:

[1148] The server preprocesses the captured audio data. The input is digital audio data, and Librosa is used for noise reduction and normalization. The output is the preprocessed audio data. Specifically, a noise filtering algorithm is applied to equalize the volume levels.

[1149] Step 3:

[1150] The server inputs pre-processed audio data into a generative AI model. The input is pre-processed audio data, which is analyzed by the generative AI model (using TensorFlow or PyTorch). The output is data for conversion into sign language or Braille. Specifically, a speech recognition algorithm analyzes the audio data and generates corresponding sign language or Braille data.

[1151] Step 4:

[1152] The server uses an emotion engine to recognize the user's emotions from audio data. The input is pre-processed audio data, and emotions are identified using OpenCV or the Emotion API. The output is the recognized emotion data. Specifically, it analyzes the tone and intonation of the voice to identify emotion categories (e.g., surprise, joy, sadness).

[1153] Step 5:

[1154] The server generates information based on recognized emotions. Input consists of emotion data and sign language or Braille data, which are combined to generate information for the user. Output is sign language or Braille information corresponding to the emotion. Specifically, it adjusts the content of the sign language or Braille based on the emotion data and sets points to emphasize.

[1155] Step 6:

[1156] The device provides users with generated sign language or Braille information. Input is sign language or Braille information corresponding to emotions, and output is information displayed on a screen or Braille display. Specifically, in the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1157] (Application Example 2)

[1158] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1159] When people with visual or hearing impairments enjoy watching sports, they often have difficulty understanding live commentary in real time. Furthermore, because information is not provided in a way that responds to the user's emotions, there is a challenge in highlighting information that reflects their changing feelings. As a result, people with visual or hearing impairments are unable to fully enjoy watching sports.

[1160] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1161] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and converting the information with emphasis into sign language or Braille. This makes it possible for visually and hearing impaired people to enjoy watching sports in real time. Furthermore, because information is emphasized according to the user's emotions, a more immersive viewing experience can be provided.

[1162] "Live broadcast audio data" refers to audio information obtained from live broadcasts of sporting events and other live events.

[1163] "Generative AI" refers to artificial intelligence technology that analyzes audio data and converts it into sign language or Braille.

[1164] Sign language is a visual gesture language used by visually impaired people to understand auditory information.

[1165] Braille is a pattern of raised dots used by visually impaired people to understand written information through touch.

[1166] An "emotion engine" is a technology that recognizes a user's emotions in real time and generates information that corresponds to those emotions.

[1167] "User emotions" refers to the emotional states, such as surprise, joy, and sadness, that users experience while watching sports.

[1168] "Highlighting information" means displaying specific information in a way that makes it stand out, depending on the user's emotions.

[1169] The system for carrying out this invention is configured as follows: The server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and converting the information with emphasis into sign language or Braille.

[1170] Program Processing Description

[1171] The server uses the smartphone's microphone to acquire live broadcast audio data. The acquired audio data is converted to text using the speech_recognition library. Next, a generative AI model is used to convert the text data into sign language or Braille. In this process, TensorFlow is used to execute the sign language and Braille conversion model.

[1172] Furthermore, the emotion engine uses the emotion_recognition library to recognize the user's emotions in real time. Based on the recognized emotions, the generative AI emphasizes specific information and converts it into sign language or Braille. This makes it possible to provide information that is tailored to the user's emotions.

[1173] Specific example

[1174] For example, if a user is watching a live broadcast of a sporting event, the commentary audio is captured through the smartphone's microphone. The audio data is converted to text in real time, and that text can then be converted into sign language or Braille. Simultaneously, an emotion engine recognizes the user's emotions, and specific scenes or information are highlighted according to those emotions, such as surprise or joy.

[1175] Example of a prompt

[1176] "This application converts live commentary from sports events into sign language and Braille in real time, providing information to help visually and hearing-impaired individuals enjoy watching sports. Furthermore, it uses an emotion engine to recognize the user's emotions and highlights information corresponding to those emotions."

[1177] In this way, people with visual or hearing impairments can enjoy watching sports in real time, and information is emphasized according to the user's emotions, providing a more immersive viewing experience.

[1178] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[1179] Step 1:

[1180] The server uses the smartphone's microphone to acquire live audio data. The input is audio data from the smartphone's microphone, and the output is an audio data file. This audio data is used in subsequent processing steps.

[1181] Step 2:

[1182] The server converts the acquired audio data into text using the speech_recognition library. The input is the audio data acquired in step 1, and the output is text data. This conversion allows the audio information to be treated as text information.

[1183] Step 3:

[1184] The server uses a generative AI model to convert text data into sign language and Braille. The input is the text data obtained in step 2, and the output is sign language data and Braille data. This conversion uses a sign language conversion model and a Braille conversion model based on TensorFlow.

[1185] Step 4:

[1186] The server uses the emotion_recognition library to recognize the user's emotions in real time. The input is the text data obtained in step 2, and the output is the user's emotion data. This emotion data is used in subsequent information enhancement processing.

[1187] Step 5:

[1188] The server converts specific information into sign language or Braille, emphasizing it based on recognized emotion data. The input is the sign language and Braille data obtained in step 3, as well as the emotion data obtained in step 4, and the output is the emphasized sign language and Braille data. This process enables the provision of information tailored to the user's emotions.

[1189] Step 6:

[1190] The server outputs highlighted sign language and Braille data to the user. The input is the highlighted sign language and Braille data obtained in step 5, and the output is the sign language and Braille information displayed on the user's device. This allows visually and hearing impaired individuals to enjoy watching sports in real time.

[1191] (Example 3)

[1192] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1193] When users with visual or hearing impairments enjoy watching sports, there is a challenge in providing them with real-time information that responds to their emotions. In particular, the lack of information that takes users' emotions into consideration limits their experience, which is a problem.

[1194] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1195] In this invention, the server includes means for acquiring user emotion data, means for analyzing said emotion data, means for generating sign language or Braille data based on the analyzed emotion data, and means for outputting said sign language or Braille data. This makes it possible to provide real-time information in response to the user's emotions.

[1196] "User emotion data" refers to data that indicates the user's emotional state, obtained from the user's facial expressions, voice, and other sources.

[1197] "Means of acquisition" refers to devices and software used to collect user emotional data.

[1198] "Means of analysis" refers to devices or software used to analyze acquired emotional data and identify the user's emotional state.

[1199] "Generating means" refers to devices and software used to generate sign language or Braille data based on analyzed emotional data.

[1200] "Output methods" refer to devices and software used to provide users with generated sign language or Braille data.

[1201] "Sign language data" refers to data that contains information expressed in sign language.

[1202] "Braille data" refers to data that contains information expressed in Braille.

[1203] "Real-time" refers to data acquisition, analysis, generation, and output occurring without delay.

[1204] "Visually impaired or hearing impaired" refers to people who have a visual or hearing impairment.

[1205] Modes for carrying out the invention

[1206] This invention is a system that provides real-time information tailored to the user's emotions, enabling users with visual or hearing impairments to enjoy watching sports. Specific embodiments of this system are described below.

[1207] System Configuration

[1208] 1. Obtaining user sentiment data

[1209] The user inputs their facial expressions and voice through the device's camera and microphone. The device transmits this data to the server in real time. The hardware used includes a camera and microphone.

[1210] 2. Analysis of emotional data

[1211] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotions the user is feeling. For example, if the user has a sad expression, the emotion engine recognizes the emotion as "sadness." The software used includes the emotion engine.

[1212] 3. Generation of sign language and braille data

[1213] The server generates sign language and Braille data using a generative AI model based on emotional data obtained from the emotion engine. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if the user expresses sadness, the generative AI model will generate sign language and Braille data that emphasizes the sad scene.

[1214] 4. Data Output

[1215] The server transmits the generated sign language data to the terminal's display and the braille data to the braille display. The terminal displays the sign language on its screen and physically forms braille on the braille display. The user can receive information by looking at the display or touching the braille display. The hardware used includes a display and a braille display.

[1216] Specific example

[1217] As a concrete example, consider a scenario where a user is watching a sports event. If the user expresses sadness, the emotion engine recognizes this emotion, and the server uses a generative AI model based on this information to generate sign language or Braille data that emphasizes the sad scene. The generated data is then provided to the user through a display or Braille display.

[1218] Example of a prompt

[1219] "When a user expresses sadness while watching a sporting event, please recognize that emotion and generate sign language or Braille data that emphasizes the sad scene."

[1220] In this way, the server can provide information tailored to the user's emotions in real time, supporting visually and hearing impaired individuals in enjoying sports viewing. The flow of specific processing in Example 3 will be explained using Figure 21.

[1221] Step 1:

[1222] The user inputs their facial expressions and voice through the device's camera and microphone. The device transmits this data to the server in real time. The input consists of the user's facial expression data and voice data, while the output is emotion data sent to the server.

[1223] Step 2:

[1224] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotions the user is feeling. The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the emotion engine detects features such as "drooping eyebrows" and "downturned corners of the mouth" and determines that the user is expressing sadness.

[1225] Step 3:

[1226] The server generates sign language and Braille data using a generative AI model based on emotional data obtained from the emotion engine. The input is the analyzed emotional information, and the output is the generated sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if the user expresses sadness, the generative AI model will generate sign language and Braille data that emphasizes the sad scene.

[1227] Step 4:

[1228] The server transmits the generated sign language data to the terminal's display and the braille data to the braille display. The input is the generated sign language data and braille data, and the output is the sign language displayed on the screen and the braille formed on the braille display. The terminal displays the sign language on the screen and physically forms the braille on the braille display. The user can understand the sign language by looking at the display and read the braille by touching the braille display.

[1229] In this way, the server can provide information tailored to the user's emotions in real time, supporting people with visual or hearing impairments in enjoying watching sports.

[1230] (Application Example 3)

[1231] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1232] When users with visual or hearing impairments enjoy watching sports, there is a challenge in providing real-time information. Furthermore, the inability to provide information tailored to the user's emotions limits their experience.

[1233] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, means for generating sign language or Braille data according to the emotions, means for displaying the sign language data on a display, and means for outputting the Braille data to a Braille display. This makes it possible for users with visual or hearing impairments to enjoy watching sports in real time, and further enables the provision of information that is tailored to the user's emotions.

[1234] "Live broadcast audio data" refers to audio data that conveys the progress of sporting events and other live events in real time.

[1235] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[1236] Sign language is a visual, gesture-based language used by people with hearing impairments to communicate.

[1237] Braille is a writing system that uses raised dots to allow visually impaired people to read by touch.

[1238] A "display" is a device used to visually display information.

[1239] A "braille display" is a device that physically creates braille characters, allowing visually impaired people to read them by touch.

[1240] "Means of recognizing user emotions" refers to technologies that analyze and recognize emotions from the user's facial expressions, voice, etc.

[1241] "Means for generating sign language data" refers to technology that converts input information into a sign language format.

[1242] "Means for generating braille data" refers to technology that converts input information into a braille format.

[1243] "Means for displaying sign language data on a screen" refers to the technology for displaying generated sign language data on a screen.

[1244] "Means for outputting braille data to a braille display" refers to the technology for outputting generated braille data to a braille display.

[1245] The system for carrying out this invention has the following configuration.

[1246] System program

[1247] The system includes means for capturing live broadcast audio data, means for analyzing the audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting sign language or Braille, means for recognizing the user's emotions, means for generating sign language or Braille data according to the emotions, means for displaying the sign language data on a display, and means for outputting the Braille data to a Braille display.

[1248] Explanation of the process

[1249] Hardware to use

[1250] Smartphone camera

[1251] Smartphone display

[1252] Braille display

[1253] Software to use

[1254] OpenCV: Capturing and displaying camera footage

[1255] EmotionRecognizer: Emotion Recognition Model

[1256] SignLanguageGenerator: Sign language generation model

[1257] BrailleGenerator: Braille generation model

[1258] Data processing and calculations

[1259] The server captures the user's face with the smartphone's camera and inputs the video into the EmotionRecognizer to recognize the user's emotions. Based on the recognized emotions, the SignLanguageGenerator generates sign language data, and the BrailleGenerator generates braille data. The generated sign language data is displayed on the smartphone's screen, and the braille data is output to a braille display.

[1260] Specific example

[1261] For example, if a user expresses sadness while watching a sporting event, the system recognizes that emotion and highlights the sad scene, converting it into sign language or Braille. It can even provide a detailed explanation in sign language or Braille of a scene where a player from the losing team is crying.

[1262] Example of a prompt

[1263] Create a program that recognizes a user's emotions and generates sign language or Braille data corresponding to those emotions. Develop an application that, when a user expresses sadness, provides information in sign language or Braille that reflects that emotion.

[1264] The above describes the embodiments for carrying out this invention. This system enables users with visual or hearing impairments to enjoy watching sports in real time, and further enables the provision of information that responds to the user's emotions.

[1265] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[1266] Step 1:

[1267] The server captures the user's face using the smartphone's camera. The input is the camera feed, and the output is the captured image data. This image data is used as preprocessing for emotion recognition.

[1268] Step 2:

[1269] The server inputs the captured image data into the EmotionRecognizer to recognize the user's emotions. The input is image data, and the output is recognized emotion data. The EmotionRecognizer extracts facial features from the image data and classifies the emotions.

[1270] Step 3:

[1271] The server generates sign language data using SignLanguageGenerator based on recognized emotion data. The input is emotion data, and the output is sign language data. SignLanguageGenerator generates sign language actions corresponding to the emotion data.

[1272] Step 4:

[1273] The server generates Braille data using BrailleGenerator based on similarly recognized emotion data. The input is emotion data, and the output is Braille data. BrailleGenerator generates Braille patterns corresponding to the emotion data.

[1274] Step 5:

[1275] The server displays the generated sign language data on the smartphone's screen. The input is the sign language data, and the output is the sign language display on the screen. The display provides the user with a visual representation of the sign language actions.

[1276] Step 6:

[1277] The server outputs the generated braille data to a braille display. The input is braille data, and the output is a braille display on the braille display. The braille display physically forms braille patterns that the user can read by touch.

[1278] Step 7:

[1279] When a user expresses an emotion while watching a sporting event, the server recognizes that emotion in real time and generates and outputs appropriate sign language or Braille data. This allows users to receive information tailored to their emotions in real time.

[1280] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1281] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1282] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1283] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1284] [Third Embodiment]

[1285] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1286] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1287] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1288] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1289] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1290] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1291] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1292] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1293] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1294] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1295] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1296] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[1297] "Example of form 1"

[1298] One embodiment of the present invention includes a microphone and audio input device for capturing audio data from live sports broadcasts. This audio input device can capture audio data from live broadcasts in real time. The captured audio data is transmitted to a generating AI.

[1299] "Example of form 2"

[1300] The generative AI analyzes captured audio data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the audio data. This generative AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[1301] "Example of form 3"

[1302] The generated sign language and braille data are provided to the user through an output device. Possible output devices include displays for showing sign language and braille displays for outputting braille. For example, a braille display physically forms braille based on the generated braille data, allowing visually impaired individuals to read it by touch.

[1303] The following describes the processing flow for each example of the form.

[1304] "Example of form 1"

[1305] Step 1: Install a microphone or audio input device to capture audio data from live sports broadcasts. This audio input device can capture audio data from live broadcasts in real time.

[1306] Step 2: The captured audio data is sent to the generating AI. This generating AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[1307] "Example of form 2"

[1308] Step 1: The generation AI analyzes the captured audio data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on the information obtained from the audio data.

[1309] Step 2: The generated sign language and braille data are provided to the user through an output device. Possible output devices include displays for showing sign language and braille displays for outputting braille.

[1310] "Example of form 3"

[1311] Step 1: As a concrete example, a braille display physically forms braille based on generated braille data, which can be read by visually impaired people through touch.

[1312] (Example 1)

[1313] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1314] There is a problem in that people with visual or hearing impairments have difficulty enjoying watching sports. In particular, they have difficulty understanding the audio information of live broadcasts in real time, which greatly diminishes their enjoyment of watching sports.

[1315] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for transmitting the live broadcast audio data to a generating AI model, and means for the generating AI model to analyze the audio data and convert it into text. This makes it possible for people with visual or hearing impairments to enjoy watching sports in real time.

[1316] "Live commentary audio data" refers to audio information provided in real time by commentators during events such as sports broadcasts.

[1317] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates a specific output.

[1318] "Means" refer to the methods or devices used to achieve a specific purpose.

[1319] "Converting to text" is the process of analyzing audio data and converting it into written information.

[1320] "Outputting" means providing processed data to the user in the form of display, audio, or other formats.

[1321] Modes for carrying out the invention

[1322] This invention is a system that captures audio data from live sports broadcasts in real time and transmits that data to a generating AI model. Specifically, it acquires audio data using a microphone or voice input device and transmits that data to a generating AI model to perform various data processing and calculations.

[1323] Hardware and software to use

[1324] Hardware: Microphone, audio input device

[1325] Software: Generative AI models (e.g., OpenAI's GPT-4)

[1326] System Overview

[1327] 1. Importing audio data:

[1328] Users provide live commentary on sports events.

[1329] Microphones and audio input devices connected to the terminal capture audio data from the live broadcast in real time.

[1330] 2. Sending audio data:

[1331] The captured audio data is sent from the terminal to the server.

[1332] The server sends the received audio data to the generating AI model.

[1333] 3. Data analysis and processing:

[1334] The generative AI model analyzes the received audio data and performs necessary data processing and calculations.

[1335] For example, it can perform tasks such as converting audio data to text, extracting important events, and generating real-time highlights.

[1336] 4. Output of results:

[1337] The server receives the results from the generated AI model and provides feedback to the user.

[1338] Specifically, the generated text is sent to the user's device, and the device displays that text.

[1339] Specific example

[1340] Specific example 1:

[1341] A user is providing live commentary on a soccer match.

[1342] The microphone picks up the voice saying "Goal!".

[1343] The device sends this audio data to the server.

[1344] The server sends audio data to the AI ​​model, which generates the text "Goal scored!".

[1345] Specific example 2:

[1346] A user is providing live commentary on a basketball game.

[1347] The microphone picks up the voice saying, "Three-point shot successful!"

[1348] The device sends this audio data to the server.

[1349] The server sends audio data to the AI ​​model, which generates the text "You made a three-point shot!".

[1350] Example of a prompt

[1351] Prompt message 1:

[1352] "Analyze this audio data and extract the important events. Audio data: 'Goal!'"

[1353] Prompt message 2:

[1354] "Please convert this audio data to text. Audio data: 'Three-point shot successful!'"

[1355] The above describes the embodiments for carrying out this invention.

[1356] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1357] Step 1:

[1358] The user provides live commentary on a sports event. A microphone and audio input device connected to the terminal capture the commentary audio data in real time. Specifically, when the user shouts "Goal!", the microphone captures the audio and converts it into digital data. The input is the user's voice, and the output is digital audio data.

[1359] Step 2:

[1360] The terminal sends the captured audio data to the server. Specifically, the terminal compresses the audio data and sends it to the server using the TCP / IP protocol. The input is digital audio data, and the output is the audio data sent to the server.

[1361] Step 3:

[1362] The server sends the received audio data to the generating AI model. Specifically, the server sends the audio data to the generating AI model's API and waits for the analysis results. The input is the audio data sent to the server, and the output is the audio data sent to the generating AI model.

[1363] Step 4:

[1364] The generative AI model analyzes the audio data and performs necessary data processing and calculations. Specifically, the generative AI model converts the audio data into text and generates the text "Goal scored!". The input is the audio data sent to the generative AI model, and the output is the generated text.

[1365] Step 5:

[1366] The server receives the results from the generated AI model and provides feedback to the user. Specifically, the server sends the generated text to the user's device, and the device displays that text. The input is the generated text, and the output is the text displayed on the user's device.

[1367] (Application Example 1)

[1368] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1369] In traditional sports viewing, it has been difficult for people with visual or hearing impairments to follow the progress of a match and follow important highlights in real time. Furthermore, general spectators have limited means of obtaining important match highlights and statistics in real time. This has resulted in a limitation in the enjoyment of watching sports.

[1370] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1371] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the transcribed live broadcast content, and means for outputting the generated highlights and statistical information. This makes it possible for people with visual or hearing impairments to obtain information in real time to enjoy watching sports, and also makes it possible for general spectators to obtain important highlights and statistical information of the match in real time.

[1372] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports and events.

[1373] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate a specific output.

[1374] Sign language is a language that uses hand and finger movements to communicate with people who are deaf or hard of hearing.

[1375] Braille is a tactile representation of writing that uses raised dots to allow visually impaired people to read text.

[1376] "Methods for converting to text" refer to technologies that analyze audio data and convert it into corresponding textual information.

[1377] A "highlight" refers to an important moment or event in a sport or event.

[1378] "Statistical information" refers to numerical data and analysis results related to matches and events.

[1379] "Means of output" refers to technologies such as display devices and audio devices that provide the generated information to the user.

[1380] A system for carrying out this invention includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the transcribed live broadcast content, and means for outputting the generated highlights and statistical information.

[1381] Hardware and software to be used

[1382] Hardware: Smartphone (with built-in microphone)

[1383] Software: Python, SpeechRecognition library (speech recognition), OpenAI API (generative AI)

[1384] Processing flow

[1385] 1. Importing audio data:

[1386] The smartphone's microphone is used to capture audio data from the live broadcast in real time. The user launches the smartphone application and begins voice input.

[1387] 2. Text conversion of audio data:

[1388] The captured audio data is converted to text using the SpeechRecognition library. The server sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data.

[1389] 3. Analysis using Generative AI:

[1390] The transcribed commentary is sent to a generating AI using the OpenAI API. The generating AI analyzes the input text data and generates important highlights and statistics.

[1391] 4. Information Output:

[1392] The generated highlights and statistics are displayed on the user's smartphone in real time. This allows users to stay informed about important moments and statistics from the match in real time.

[1393] Specific example

[1394] When a user is watching a soccer match, they launch a smartphone app to capture the audio from the live broadcast. The app converts the audio into text and displays important events such as "Goal!" and "Foul!" in real time. Furthermore, it automatically generates and displays match highlights and statistics (e.g., number of shots, ball possession).

[1395] Example of a prompt

[1396] Please generate the following sports commentary highlights:

[1397] "Team A scored a goal in the 30th minute of the first half. Team B's defense broke down. Team A continued their offensive in the second half as well."

[1398] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1399] Step 1:

[1400] The user launches a smartphone application to capture audio from a live broadcast. The user uses the smartphone's microphone to collect audio data in real time. The input is the audio data from the live broadcast, and the output is a stream of audio data.

[1401] Step 2:

[1402] The device converts the captured audio data into text using the SpeechRecognition library. The device then sends the audio data to the Google Speech Recognition API to retrieve the corresponding text data. The input is a stream of audio data, and the output is text data.

[1403] Step 3:

[1404] The server sends the transcribed commentary to the generating AI using the OpenAI API. The server analyzes the input text data and generates important highlights and statistics. The input is text data, and the output is the generated highlights and statistics.

[1405] Step 4:

[1406] The server sends the generated highlights and statistics to the user's smartphone in real time. The user can view this information on their smartphone screen. The input is the generated highlights and statistics, and the output is the information displayed on the smartphone screen.

[1407] Step 5:

[1408] Users can access important moments and statistics from a match in real time through a smartphone application. Specifically, users check the information on the screen and understand the progress of the match. The input is the information displayed on the smartphone screen, and the output is the user's understanding.

[1409] (Example 2)

[1410] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1411] There is a problem in that people with visual or hearing impairments have difficulty understanding auditory information in real time. In particular, the accuracy and speed of converting auditory information into sign language or Braille are insufficient, which can lead to delays in information transmission and misunderstandings. This creates a challenge in that it is difficult for people with visual or hearing impairments to receive information accurately and quickly.

[1412] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1413] In this invention, the server includes means for acquiring audio data, means for converting the audio data into text data using speech recognition software, and means for inputting the text data into a generation AI model to generate sign language data or Braille data. This makes it possible for visually and hearing impaired people to receive audio information in real time as sign language or Braille.

[1414] "Audio data" refers to data that records audio in digital format.

[1415] "Speech recognition software" is software that analyzes speech data and converts it into text data.

[1416] "Text data" refers to data that records character information in a digital format.

[1417] A "generative AI model" is an artificial intelligence model that is trained using deep learning technology to generate sign language data or Braille data based on text data.

[1418] "Sign language data" refers to data that records sign language movements in digital format.

[1419] "Braille data" refers to data that records Braille information in digital format.

[1420] "Sign language animation software" is software that visually displays sign language actions based on sign language data.

[1421] A "braille display" is a device that physically displays braille based on braille data.

[1422] This invention is a system that enables people with visual or hearing impairments to receive audio information in real time as sign language or Braille. Specific embodiments of this system are described below.

[1423] System Configuration

[1424] Server Configuration

[1425] The server includes means for acquiring audio data, means for converting audio data into text data using speech recognition software, and means for converting text data into sign language data or Braille data using a generative AI model. For speech recognition software, a common speech recognition API (e.g., Google Cloud Speech-to-Text API) is used. The generative AI model is a model trained using deep learning techniques (e.g., a model using TensorFlow or PyTorch).

[1426] Device configuration

[1427] The terminal includes means for visually displaying sign language data transmitted from the server using sign language animation software (e.g., SignAll), and means for outputting braille data to a braille display (e.g., Focus Blue). The terminal also includes a microphone for the user to input voice data.

[1428] User actions

[1429] The user inputs voice data by speaking into the device's microphone. For example, they might say "hello." The device sends this voice data to a server. The server converts the voice data into text data and uses a generative AI model to generate sign language data or Braille data. The generated data is sent to the device and displayed as a sign language animation on the screen or output to a Braille display.

[1430] Specific example

[1431] Example 1: Conversion to sign language

[1432] When a user says "Good morning," the device sends the audio data to a server. The server converts the audio data into text and inputs the text information "Good morning" into a generative AI model. The generative AI model generates sign language data corresponding to "Good morning" and creates a sign language animation using sign language animation software. The device displays this sign language animation on its screen, allowing the user to visually confirm the sign language for "Good morning."

[1433] Example 2: Conversion to Braille

[1434] When the user says "thank you," the device sends the audio data to a server. The server converts the audio data into text and inputs the text information "thank you" into a generating AI model. The generating AI model generates braille data corresponding to "thank you" and sends it to a braille display. The braille display connected to the device displays "thank you" in braille, and the user can read the braille by touch.

[1435] Example of a prompt

[1436] "Please explain how the generative AI model translates a user's 'Good morning' into sign language or Braille."

[1437] By inputting this prompt into the generating AI model, a detailed explanation of the specific transformation process can be obtained.

[1438] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1439] Step 1:

[1440] The user inputs voice data.

[1441] The user speaks into the device's microphone. For example, they might say "Hello." The device captures this voice as digital audio data. The input is the user's voice, and the output is digital audio data.

[1442] Step 2:

[1443] The device sends voice data to the server.

[1444] The device transmits the captured audio data to the server via the internet. During this process, the data is encoded in an appropriate format (e.g., WAV or MP3). The input is digital audio data, and the output is the audio data transmitted to the server.

[1445] Step 3:

[1446] The server converts the audio data into text data.

[1447] The server converts the received audio data into text data using speech recognition software (e.g., a common speech recognition API). For example, the audio data "Hello" is converted into the text data "Hello". The input is audio data, and the output is text data.

[1448] Step 4:

[1449] The server generates text data and inputs it into the AI ​​model.

[1450] The server inputs the converted text data into a generative AI model. This generative AI model is trained using deep learning techniques. The input is text data, and the output is sign language data or Braille data.

[1451] Step 5:

[1452] The generative AI model generates sign language data and Braille data.

[1453] The generative AI model generates sign language data and Braille data based on input text data. For example, the text data "Hello" is converted into sign language animation data and Braille data. The input is text data, and the output is sign language data and Braille data.

[1454] Step 6:

[1455] The server sends the generated data to the terminal.

[1456] The server sends the generated sign language or braille data to the terminal. The data is encoded in an appropriate format (e.g., JSON or XML). The input is the sign language or braille data, and the output is the data sent to the terminal.

[1457] Step 7:

[1458] The terminal displays sign language animations and outputs Braille data to a Braille display.

[1459] The terminal displays received sign language data on its screen using sign language animation software (e.g., sign language animation software). Braille data is output to a braille display connected to the terminal (e.g., a braille display). The user can visually confirm the sign language animation displayed on the screen and tactilely read the braille displayed on the braille display. Input is sign language data and braille data, and output is sign language animation and braille.

[1460] (Application Example 2)

[1461] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1462] There is a problem in that visually impaired and hearing-impaired individuals have difficulty understanding voice instructions and warnings in real time during security work. This can prevent them from performing their duties efficiently, potentially leading to a decline in the quality of security.

[1463] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the audio data into text, means for converting the text into sign language and Braille, and means for providing the sign language and Braille to visually impaired and hearing impaired people. This makes it possible for visually impaired and hearing impaired people to understand voice instructions and warnings in real time and perform security work efficiently.

[1464] "Live broadcast audio data" refers to audio information of events and situations that are taking place in real time.

[1465] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[1466] "Sign language" refers to a means of conveying linguistic information using visual gestures and movements.

[1467] Braille refers to a writing system that uses raised dots to allow visually impaired people to read text information using their sense of touch.

[1468] "Audio data" refers to information recorded in digital format using audio.

[1469] "Text" refers to written information.

[1470] "Visually impaired" refers to people who have a visual impairment.

[1471] "People with hearing impairments" refers to individuals who have a hearing loss.

[1472] "Security work" refers to work performed to protect the safety of facilities and people.

[1473] "Real-time" refers to processing or communication occurring instantly without delay.

[1474] The system for carrying out this invention converts audio data into sign language or Braille in real time and provides it to visually impaired and hearing impaired individuals. Specific embodiments of the system are described below.

[1475] System Configuration

[1476] hardware

[1477] Smartphone: Equipped with a microphone for acquiring audio data and a display for showing the conversion results.

[1478] Server: Provides computing resources for analyzing and converting audio data.

[1479] software

[1480] Speech recognition library: Use the speech_recognition library to convert speech data to text.

[1481] Sign language conversion library: Use the hand_gesture_recognition library to convert text into sign language.

[1482] Braille conversion library: Use the braille_converter library to convert text to Braille.

[1483] Generative AI Model: Uses a deep learning model to analyze audio data and convert it into sign language or Braille.

[1484] System operation

[1485] Acquisition of audio data

[1486] The user inputs voice data using their smartphone's microphone. For example, a security guard might input "There is a suspicious person" by voice.

[1487] Analysis of audio data

[1488] The server converts the acquired audio data into text using the speech_recognition library. This text forms the basis for subsequent sign language and Braille conversion.

[1489] Conversion to sign language and Braille

[1490] The server uses a generative AI model to convert text into sign language and Braille. The `hand_gesture_recognition` library is used for sign language conversion, and the `braille_converter` library is used for Braille conversion.

[1491] Providing results

[1492] The converted sign language and Braille are displayed on the smartphone screen. This allows visually impaired and hearing-impaired individuals to understand voice instructions and warnings in real time.

[1493] Specific example

[1494] For example, if a security guard voice-inputs "There is a suspicious person," the system converts this voice into text, and then into sign language and Braille. The results are displayed on a smartphone screen, ensuring that the information is also conveyed to security guards who are visually impaired or hearing impaired.

[1495] Example of a prompt

[1496] Please enter the audio data. Example: "There is a suspicious person."

[1497] In this way, visually impaired and hearing-impaired individuals can understand voice instructions and warnings in real time, enabling them to perform security tasks efficiently.

[1498] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1499] Step 1:

[1500] The user inputs voice data using the smartphone's microphone. The input voice data is temporarily stored in the smartphone's internal memory.

[1501] Step 2:

[1502] The smartphone sends the stored audio data to the server. The server receives the audio data and uses the speech_recognition library to convert it to text. The input is audio data, and the output is text data.

[1503] Step 3:

[1504] The server analyzes text data using a generative AI model. This analysis converts the text data into sign language and Braille. The input is text data, and the output is sign language data and Braille data.

[1505] Step 4:

[1506] The server converts sign language data into sign language using the hand_gesture_recognition library. The input is text data, and the output is sign language data.

[1507] Step 5:

[1508] The server converts braille data into braille using the braille_converter library. The input is text data, and the output is braille data.

[1509] Step 6:

[1510] The server sends the converted sign language and Braille data to the smartphone. The smartphone displays the received sign language and Braille data on its screen. This allows visually impaired and hearing-impaired individuals to understand voice instructions and warnings in real time.

[1511] (Example 3)

[1512] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1513] The aim is to solve the problem that users with visual or hearing impairments have difficulty converting text data into sign language or Braille and accessing information in real time. In particular, there is a need for a system that can quickly and accurately convert text data entered by users into sign language or Braille and provide it through an appropriate output device.

[1514] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1515] In this invention, the server includes means for a user to input text data, means for the server to receive the input data and send prompt sentences to a generating AI model, means for the generating AI model to generate sign language or Braille data, means for the server to transmit the generated data to an output device, and means for the terminal to provide sign language or Braille to the user through the output device. This makes it possible for users with visual or hearing impairments to quickly and accurately convert the input text data into sign language or Braille and use the information in real time.

[1516] A "user" refers to a person who uses the system to input text data and receives the information converted into sign language or Braille.

[1517] A "server" refers to a computer system that receives input data from users, sends prompt messages to the AI ​​generation model, and transmits the generated sign language or Braille data to an output device.

[1518] A "generative AI model" refers to an artificial intelligence model that generates sign language or Braille data based on text data from users.

[1519] A "prompt sentence" refers to a sentence used to instruct a generative AI model to convert text data into sign language or Braille.

[1520] An "output device" is a device that provides generated sign language or Braille data to the user, and includes sign language displays and Braille displays.

[1521] A "sign language display" refers to a display device used to visually display generated sign language data.

[1522] A "braille display" refers to a device that physically forms generated braille data, allowing visually impaired individuals to read it by touch.

[1523] "Text data" refers to the character information that a user enters into the system.

[1524] "Real-time" refers to the process where text data entered by a user is instantly converted into sign language or Braille and provided almost simultaneously through an output device.

[1525] Modes for carrying out the invention

[1526] This invention is a system that enables users with visual or hearing impairments to convert text data into sign language or Braille and access information in real time. Specific embodiments of this system are described below.

[1527] System Configuration

[1528] This system consists of the following main components:

[1529] 1. User terminal: A device used by a user to input text data. Examples include personal computers, smartphones, and tablets.

[1530] 2. Server: This server receives input data from the user, sends prompt messages to the generation AI model, and sends the generated sign language or Braille data to the output device. A high-performance server (e.g., a server equipped with an NVIDIA GPU) should be used.

[1531] 3. Generative AI Model: An artificial intelligence model for generating sign language or Braille data based on text data from users. It uses deep learning frameworks such as TensorFlow or PyTorch.

[1532] 4. Output device: A device for providing the generated sign language or Braille data to the user. This includes sign language displays and Braille displays.

[1533] System operation

[1534] 1. The user enters text data.

[1535] Users input text data through the system interface. For example, they might type "Hello" using a web browser or a dedicated application.

[1536] 2. The server receives the input data and sends a prompt message to the generating AI model.

[1537] The server receives text data sent by the user. Based on the received data, it creates and sends a prompt to the generative AI model. For example, it might send the prompt "Please convert the following text into sign language: Hello" to the generative AI model.

[1538] 3. The generative AI model generates data for sign language and Braille.

[1539] The generative AI model analyzes prompt messages received from the server and generates corresponding sign language or Braille data. For example, it can generate sign language animation data or Braille sequence data.

[1540] 4. The server sends the generated data to the output device.

[1541] The server transmits the sign language and Braille data received from the generated AI model to the output devices. For example, it transmits sign language animation data to a sign language display and Braille data to a Braille display.

[1542] 5. The terminal provides sign language or Braille to the user through an output device.

[1543] The terminal receives data transmitted from the server and provides it to the user through an output device. The sign language display shows sign language animations, and the braille display physically forms braille characters. Users can visually confirm the sign language display and tactilely read the braille display.

[1544] Specific example

[1545] For example, the specific actions taken when a user enters the text "thank you" are as follows:

[1546] 1. User: Type "Thank you" into the input field of the web browser and click the submit button.

[1547] 2. Server: Receives the text data "thank you" from the user and sends the prompt message "Please convert the following text into sign language: thank you" to the AI ​​model.

[1548] 3. Generative AI Model: Analyzes the prompt sentence and generates sign language animation data corresponding to "thank you".

[1549] 4. Server: Sends the generated sign language animation data to the sign language display.

[1550] 5. Terminal: The sign language display shows sign language animations for the user to visually confirm.

[1551] In this way, users can easily convert text into sign language or Braille and use it through an output device. The flow of the specific processing in Example 3 will be explained with reference to Figure 15.

[1552] Step 1:

[1553] The user enters text data.

[1554] Users input text data using a web browser or a dedicated application. For example, they might type "Hello" and click the submit button. The input data is then sent from the user's terminal to the server.

[1555] Step 2:

[1556] The server receives the input data and sends prompt messages to the generating AI model.

[1557] The server receives text data sent by the user. Based on the received data, it creates and sends a prompt to the generative AI model. For example, it might send the prompt "Please convert the following text into sign language: Hello" to the generative AI model. The input is the user's text data, and the output is the prompt to the generative AI model.

[1558] Step 3:

[1559] The generative AI model generates data for sign language and Braille.

[1560] The generative AI model analyzes prompt text received from the server and generates corresponding sign language or Braille data. For example, it can generate sign language animation data or Braille sequence data. The input is prompt text, and the output is sign language or Braille data.

[1561] Step 4:

[1562] The server sends the generated data to the output device.

[1563] The server transmits the sign language and Braille data received from the generating AI model to the output device. For example, it transmits sign language animation data to a sign language display and Braille data to a Braille display. The input is the generated sign language and Braille data, and the output is the transmission of data to the output device.

[1564] Step 5:

[1565] The terminal provides sign language and Braille to the user through an output device.

[1566] The terminal receives data transmitted from the server and provides it to the user through an output device. The sign language display shows sign language animations, and the braille display physically forms braille characters. The user can visually confirm the sign language display and tactilely read the braille display. The input is sign language or braille data from the server, and the output is the provision of sign language or braille to the user.

[1567] (Application Example 3)

[1568] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1569] There is a problem in that people with visual or hearing impairments have difficulty receiving real-time information when watching sports or obtaining security information. Furthermore, there is a need for rapid information transmission in sign language and Braille during emergencies, but the current system is insufficient to meet this need.

[1570] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for acquiring video from a security camera, analyzing it with a sign language recognition model and recognizing sign language, means for displaying the recognized sign language on the video, and means for outputting security information to a Braille display. This makes it possible for people with visual or hearing impairments to watch sports and receive security information in real time.

[1571] "Live broadcast audio data" refers to data that conveys the progress of sporting events and other live events through audio.

[1572] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[1573] Sign language is a visual, gesture-based language used by people with hearing impairments to communicate.

[1574] Braille is a writing system that uses raised dots to allow visually impaired people to read by touch.

[1575] A "sign language recognition model" is a machine learning model that analyzes sign language movements from camera footage and other sources and converts them into the corresponding language.

[1576] A "security camera" is a camera device used to monitor a specific area and acquire video data.

[1577] A "Braille display" is a device that displays Braille using electronically controlled raised dots.

[1578] "Security information" refers to information regarding anomalies or emergencies within the monitored area.

[1579] The system for carrying out this invention is configured as follows: The server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for acquiring video from a security camera, analyzing it with a sign language recognition model and recognizing sign language, means for displaying the recognized sign language on the video, and means for outputting security information to a Braille display.

[1580] Hardware and software to be used

[1581] Hardware:

[1582] Smartphone camera

[1583] Braille display (e.g., BrailleDisplay)

[1584] Security camera

[1585] software:

[1586] OpenCV: Acquiring and displaying camera images

[1587] HandSignRecognition: Sign Language Recognition Model

[1588] Braille Display: Braille display control library

[1589] Data processing and data calculation

[1590] 1. Importing live broadcast audio data:

[1591] The server receives audio data from live broadcasts that provide updates on the progress of sporting events and other live events.

[1592] 2. Analysis using Generative AI:

[1593] The server uses AI to analyze the captured live broadcast audio data and converts it into sign language or Braille in real time.

[1594] 3. Output of sign language and Braille:

[1595] The server outputs the generated sign language or Braille through a display or Braille display.

[1596] 4. Capture video from security cameras:

[1597] The server captures video from security cameras and analyzes it using a sign language recognition model.

[1598] 5. Recognition and display of sign language:

[1599] The server uses a sign language recognition model to recognize sign language in the video and displays the recognized sign language on the video.

[1600] 6. Outputting security information to a Braille display:

[1601] The server outputs security information to a braille display.

[1602] Specific example

[1603] Watching sports:

[1604] Users with visual or hearing impairments can receive information via displays or braille displays by having live audio data from sports events converted into sign language or braille in real time.

[1605] Obtaining security information:

[1606] Security cameras capture video footage, a sign language recognition model recognizes the sign language, and the recognized sign language is displayed on the video. Additionally, security information is output to a Braille display, allowing visually impaired individuals to receive information through touch.

[1607] Example of a prompt

[1608] "Create a program that analyzes security camera footage, displays emergency information in sign language, and outputs security information to a Braille display."

[1609] The above describes the embodiments for carrying out this invention.

[1610] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[1611] Step 1:

[1612] The server receives audio data from live broadcasts of sporting events and other live events, providing commentary on the progress of those events. The input is the live broadcast audio data, and the output is the captured audio data. This audio data is used for subsequent analysis.

[1613] Step 2:

[1614] The server analyzes the incoming live broadcast audio data using a generating AI and converts it into sign language and Braille in real time. The input is live broadcast audio data, and the output is sign language data and Braille data. The generating AI analyzes the audio data and generates the corresponding sign language and Braille.

[1615] Step 3:

[1616] The server outputs the generated sign language and Braille through a display or Braille display. Input is sign language data and Braille data, and output is sign language display on the display and Braille display on the Braille display. This allows users with visual or hearing impairments to receive information.

[1617] Step 4:

[1618] The server captures video from security cameras and analyzes it using a sign language recognition model. The input is video data from the security cameras, and the output is recognized sign language data. The sign language recognition model analyzes the video data and recognizes sign language movements.

[1619] Step 5:

[1620] The server displays the recognized sign language on the video. The input is the recognized sign language data, and the output is the sign language displayed on the video. This allows the user to understand security camera footage in sign language.

[1621] Step 6:

[1622] The server outputs security information to a braille display. The input is security information, and the output is a braille display on the braille display. This allows the user to receive security information through touch.

[1623] The above are the specific processing steps for carrying out this invention.

[1624] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1625] "Example of form 1"

[1626] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes the user's emotions and generates information to help visually or hearing-impaired individuals enjoy watching sports more. Specifically, it estimates emotions from the user's facial expressions, tone of voice, and word choice, and analyzes live broadcast audio data according to those emotions. For example, if the user is showing emotion of joy, the system recognizes that emotion and highlights more exciting scenes, converting them into sign language or Braille.

[1627] "Example of form 2"

[1628] Furthermore, the emotion engine recognizes emotions in real time and generates information accordingly. This makes it possible to provide information that allows visually and hearing-impaired individuals to enjoy watching sports in accordance with the emotions the user is feeling in real time. Specifically, if a user expresses surprise, the system recognizes that emotion, emphasizes the surprising scene, and converts it into sign language or Braille.

[1629] "Example of form 3"

[1630] Furthermore, the emotion engine generates information in response to the user's emotions and outputs that information in real time. This makes it possible to instantly provide information that allows visually and hearing-impaired individuals to enjoy watching sports, tailored to the emotions the user is feeling. Specifically, if a user expresses sadness, the system recognizes that emotion, emphasizes the sad scene, and converts it into sign language or Braille.

[1631] The following describes the processing flow for each example of the form.

[1632] "Example of form 1"

[1633] Step 1: Estimate the user's emotions from their facial expressions, tone of voice, and word choice.

[1634] Step 2: Analyze the live commentary audio data based on the estimated emotions.

[1635] Step 3: Based on the analysis results, highlight the most exciting scenes and convert them into sign language or Braille.

[1636] "Example of form 2"

[1637] Step 1: Recognize the user's emotions in real time.

[1638] Step 2: Analyze the live commentary audio data based on the recognized emotions.

[1639] Step 3: Based on the analysis results, highlight the surprising scenes and convert them into sign language or Braille.

[1640] "Example of form 3"

[1641] Step 1: Recognize the user's emotions in real time and generate information in response to those emotions.

[1642] Step 2: Output the generated information in real time.

[1643] Step 3: Based on the output information, emphasize the sad scenes and convert them into sign language or Braille.

[1644] (Example 1)

[1645] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1646] When users with visual or hearing impairments enjoy watching sports, they face the challenge of understanding the audio information in live broadcasts in real time. Furthermore, the inability to provide information that aligns with the user's emotions means that the enjoyment of watching the game is not fully conveyed.

[1647] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generation AI model and recognizing the user's emotions, and means for converting the live broadcast audio data into sign language or Braille in real time according to the user's emotions. This makes it possible to provide real-time information for users with visual or hearing impairments to enjoy watching sports.

[1648] "Live commentary audio data" refers to data that transmits the progress of sporting events and other live events in real time via audio.

[1649] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to analyze data and generate information tailored to a specific purpose.

[1650] "Means of recognizing user emotions" refer to technologies and devices that analyze a user's facial expressions, tone of voice, word choice, etc., to estimate their emotional state.

[1651] "Means of converting to sign language or Braille" refer to software and hardware for converting audio data or text data into sign language videos or Braille information.

[1652] "Means of outputting sign language or Braille" refer to output devices such as displays and Braille displays that provide users with generated sign language videos and Braille information.

[1653] An "exciting scene" refers to a particularly important moment or play in a sporting event that evokes excitement or emotion.

[1654] This invention is a system for users with visual or hearing impairments to enjoy watching sports. This system captures live broadcast audio data in real time, recognizes the user's emotions, and provides information corresponding to those emotions in sign language or Braille.

[1655] Hardware and software to use

[1656] hardware

[1657] Microphones: Installed in stadiums and event venues to capture audio data of live broadcasts in real time.

[1658] Camera: Used to capture the user's facial expressions.

[1659] Terminal: A device used to process audio and video data and send it to a server.

[1660] Braille display: An output device for providing Braille information to users.

[1661] software

[1662] Generative AI model: An algorithm that analyzes voice data and recognizes the user's emotions.

[1663] Emotion engine: Software used to estimate emotions from a user's facial expressions, tone of voice, word choice, etc.

[1664] Sign language translation software: Software used to convert audio or text data into sign language videos.

[1665] Braille conversion software: Software used to convert audio data and text data into Braille information.

[1666] Specific operation of the system

[1667] Importing audio data

[1668] The device uses high-sensitivity microphones installed in the stadium to capture audio data from the live broadcast in real time. For example, it can capture the audio at the moment the commentator shouts "Goal!".

[1669] Sending audio data

[1670] The device transmits the captured audio data to a generative AI model in the cloud via the internet. Specifically, the audio data is compressed and sent to the server using a secure communication protocol.

[1671] User emotion recognition

[1672] The server monitors whether the user is connected to the system via camera and microphone. The emotion engine analyzes the user's facial expressions (e.g., smiles or expressions of surprise) and voice tone (e.g., excited voice) in real time to estimate the user's emotions.

[1673] Emotion-based data analysis

[1674] If the server estimates that the user's emotion is one of joy, it will highlight exciting scenes from the live commentary audio data. For example, it will identify and extract moments of goals or important plays.

[1675] Information generation and provision

[1676] The server converts the extracted exciting scenes into sign language and Braille. Specifically, it uses sign language translation software to generate sign language videos and Braille conversion software to display them on a Braille display. The generated information is sent to the user's terminal and displayed in real time.

[1677] Specific examples and prompt statements

[1678] Specific example

[1679] Let's say a user is watching a soccer match. If the user shows an expression of joy during a goal, the system recognizes that emotion and emphasizes the commentary about the goal, converting it into sign language or Braille.

[1680] Example of a prompt

[1681] "Please capture the live commentary audio of a goal scored during a soccer match in real time, recognize the user's emotions of joy, and convert that scene into sign language or Braille."

[1682] In this way, the system provides information tailored to the user's emotions, enabling people with visual or hearing impairments to enjoy watching sports more.

[1683] The flow of the specific processing in Example 1 will be explained using Figure 17.

[1684] Step 1:

[1685] The device uses high-sensitivity microphones installed in the stadium to capture audio data from the live broadcast in real time. The input is the audio data from the live broadcast, and the output is the captured audio data. Specifically, it uses a noise-canceling function to remove background noise.

[1686] Step 2:

[1687] The device transmits the captured audio data to a generative AI model in the cloud via the internet. The input is the captured audio data, and the output is the audio data sent to the generative AI model. Specifically, the audio data is compressed and sent to the server using a secure communication protocol.

[1688] Step 3:

[1689] The server monitors the user's connection to the system via camera and microphone. Inputs are the user's facial expression data and voice tone data, and outputs are the data sent to the emotion engine. Specifically, the camera captures the user's facial expressions, and the microphone collects the user's voice tone.

[1690] Step 4:

[1691] The server uses an emotion engine to estimate the user's emotions from their facial expressions, tone of voice, and word choice. The input is the user's facial expression data and tone of voice data, and the output is the estimated emotion data. Specifically, the emotion engine analyzes the data in real time to estimate the user's emotions.

[1692] Step 5:

[1693] If the server estimates the user's emotion to be joy, it highlights exciting scenes from the live commentary audio data. The input is the estimated emotion data and the live commentary audio data, and the output is the data of the highlighted exciting scenes. Specifically, it uses an algorithm to detect certain keywords (e.g., "goal," "great play") contained in the live commentary audio data.

[1694] Step 6:

[1695] The server converts the extracted exciting scenes into sign language or Braille. The input is data of the highlighted exciting scenes, and the output is sign language video or Braille information. Specifically, it uses sign language translation software and Braille conversion software to convert the data.

[1696] Step 7:

[1697] The server transmits the generated sign language video and Braille information to the user's terminal. The input is the sign language video and Braille information, and the output is the sign language video and Braille information displayed on the user's terminal. Specifically, a low-latency communication protocol is used to ensure real-time operation.

[1698] In this way, the system provides information tailored to the user's emotions, enabling people with visual or hearing impairments to enjoy watching sports more.

[1699] (Application Example 1)

[1700] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1701] When users with visual or hearing impairments enjoy watching sports, simply converting the audio data of live broadcasts into sign language or Braille in real time presents a challenge in providing information that aligns with the user's emotions. Furthermore, the lack of emphasis on exciting scenes that take the user's emotions into consideration makes it difficult for visually and hearing-impaired individuals to fully enjoy watching sports.

[1702] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1703] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, and means for highlighting exciting scenes according to those emotions. This makes it possible to provide information that responds to the user's emotions so that visually and hearing impaired people can enjoy watching sports more.

[1704] "Live broadcast audio data" refers to audio information that conveys the progress of sporting events and other live events in real time.

[1705] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate a specific output.

[1706] Sign language is a means of conveying linguistic information using visual gestures and movements.

[1707] Braille is a pattern of dots that are raised on paper or other surfaces so that visually impaired people can read it using their sense of touch.

[1708] "Means of recognizing user emotions" refers to technologies that estimate emotions from a user's facial expressions, tone of voice, word choice, etc.

[1709] "Methods for emphasizing exciting scenes" refers to techniques for selecting particularly exciting moments in sports viewing and emphasizing those moments to people with visual or hearing impairments.

[1710] The system for carrying out this invention is configured as follows: First, the server is equipped with a microphone and an audio input device for capturing live broadcast audio data. This audio input device can capture live broadcast audio data of sports events in real time.

[1711] Next, the captured audio data is sent to a generating AI. The generating AI analyzes the audio data and converts it into sign language or Braille in real time. This converted sign language or Braille data is output in a format that can be understood by people with visual or hearing impairments.

[1712] Furthermore, the system is equipped with an emotion recognition engine to recognize the user's emotions. This engine estimates emotions from the user's facial expressions, tone of voice, and word choice. The emotion recognition engine acquires and analyzes the user's emotional data using a camera and microphone.

[1713] When a user's emotions are recognized, the system highlights exciting scenes accordingly. For example, if a user expresses joy, the system recognizes this emotion and selects and highlights particularly exciting scenes. This highlighted information is then translated into sign language or Braille and provided to people with visual or hearing impairments.

[1714] The specific hardware used includes a microphone, camera, and server. The software used includes Python, the speech_recognition library, the transformers library, and the opencv library. Combining this hardware and software enables the capture of speech data, emotion recognition, conversion to sign language or Braille, and highlighting of exciting scenes.

[1715] For example, if a user expresses feelings of joy while watching a sporting event, the system recognizes those feelings and translates them into sign language or Braille, highlighting particularly exciting scenes. This information is provided to the user in real time.

[1716] Examples of prompt statements to input into a generative AI model include the following:

[1717] If a user is expressing feelings of joy, how can I highlight the exciting scenes and translate them into sign language or Braille?

[1718] This prompt can be used to query the generative AI model for specific sign language or Braille conversion methods.

[1719] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[1720] Step 1:

[1721] The server uses a microphone to capture live commentary audio data in real time.

[1722] Input: Live broadcast audio data

[1723] Output: Captured audio data

[1724] Specific operation: The server captures the audio signal from the microphone as digital data and performs preprocessing for speech recognition.

[1725] Step 2:

[1726] The server sends the captured audio data to the AI ​​for analysis.

[1727] Input: Captured audio data

[1728] Output: Analyzed audio data

[1729] Specific operation: The server inputs the audio data into a generation AI model, which then performs speech recognition and natural language processing. This converts the audio data into text data.

[1730] Step 3:

[1731] The server converts the analyzed audio data into sign language or Braille in real time.

[1732] Input: Analyzed audio data

[1733] Output: Sign language data, Braille data

[1734] Specific operation: The server executes an algorithm that converts text data into sign language or Braille using a generative AI model. The sign language data is output in video format, and the Braille data is output in a data format for tactile displays.

[1735] Step 4:

[1736] The server uses a camera to capture the user's facial expressions and sends them to the emotion recognition engine.

[1737] Input: User's facial expression data

[1738] Output: Recognized emotion data

[1739] Specific operation: The server acquires video data from the camera and inputs it into the emotion recognition engine. The emotion recognition engine performs facial expression analysis and estimates the user's emotions.

[1740] Step 5:

[1741] The server highlights exciting scenes based on recognized emotion data.

[1742] Input: Recognized emotion data, sign language data, braille data

[1743] Output: Highlighted sign language data, Highlighted braille data

[1744] Specific operation: The server analyzes emotional data and, if the user is expressing feelings of joy, selects and highlights particularly exciting scenes. This adjusts the sign language and braille data according to the user's emotions.

[1745] Step 6:

[1746] The server provides users with highlighted sign language and Braille data.

[1747] Input: Highlighted sign language data, Highlighted Braille data

[1748] Output: Sign language data and Braille data provided to the user.

[1749] Specific operation: The server sends highlighted sign language data to the user in video format and braille data in a data format for tactile displays. The user receives this data in real time and can enjoy watching sports.

[1750] (Example 2)

[1751] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1752] When users with visual or hearing impairments enjoy watching sports, they face the challenge of understanding the audio information in live broadcasts in real time. Furthermore, because information is not provided in a way that responds to the user's emotions, it is difficult to share the sense of presence and emotion.

[1753] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1754] In this invention, the server includes means for acquiring live broadcast audio data, means for preprocessing the live broadcast audio data, means for analyzing the preprocessed audio data using a generation AI model and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions from the audio data, and means for generating information based on the recognized emotions. This makes it possible for users with visual or hearing impairments to enjoy watching sports in real time.

[1755] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports events and other events.

[1756] "Preprocessing" refers to processes such as noise reduction and normalization performed to convert audio data into a format that is easier to analyze.

[1757] A "generative AI model" is an artificial intelligence model trained using deep learning technology to analyze speech data and convert it into sign language or Braille.

[1758] Sign language is a visual gesture language used by visually impaired people to understand auditory information.

[1759] Braille is a writing system that uses raised dots to allow visually impaired people to understand written information through touch.

[1760] "Emotion recognition" is an analytical technique used to identify a user's emotions from voice data.

[1761] "Information generation" is the process of creating information to provide to the user based on analyzed voice data and recognized emotions.

[1762] "Output means" refers to devices or interfaces for providing generated sign language or Braille information to the user.

[1763] This invention is a system for users with visual or hearing impairments to enjoy watching sports in real time. Specific embodiments of this system are described below.

[1764] System Configuration

[1765] hardware

[1766] The server uses a microphone (e.g., a standard microphone) to capture audio data. It also has a high-performance processor and memory for processing and analyzing the audio data. Furthermore, it is equipped with a display and a braille display for outputting sign language and braille information.

[1767] software

[1768] The server uses Librosa, a Python library, to preprocess audio data. The generative AI model is built using TensorFlow and PyTorch, and analyzes audio data to convert it into sign language or Braille. OpenCV and the Emotion API are used for emotion recognition.

[1769] System operation

[1770] The server captures live broadcast audio data in real time via a microphone. The captured audio data is preprocessed using Librosa, including noise reduction and normalization. The preprocessed audio data is then input into a generative AI model and converted into sign language or Braille.

[1771] Simultaneously, the server uses an emotion engine to recognize the user's emotions in real time from the audio data. Based on the recognized emotions, information tailored to the user's feelings is generated. For example, if the user expresses surprise, the system recognizes that emotion and highlights the surprising scene, converting it into sign language or Braille.

[1772] The generated sign language and Braille information is provided to the user via the terminal. In the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1773] Specific example

[1774] If a user exclaims "Amazing!" while watching a sporting event, the server captures the audio and inputs it into a generative AI model. The generative AI model analyzes the audio and generates data that converts the exclamation "Amazing!" into sign language or Braille. Simultaneously, an emotion engine recognizes the user's emotion of surprise and generates information that emphasizes the surprising scene according to that emotion.

[1775] Example of a prompt

[1776] "Please describe a system that, when a user exclaims in surprise while watching a sporting event, converts that sound into sign language or Braille in real time, highlighting the surprising moment and providing information to people with visual or hearing impairments."

[1777] This system allows users with visual or hearing impairments to enjoy watching sports in real time.

[1778] The flow of the specific processing in Example 2 will be explained using Figure 19.

[1779] Step 1:

[1780] The server captures live broadcast audio data in real time via a microphone. The input is analog audio data, which is converted to a digital format and stored in temporary storage. Specifically, the audio input device captures the audio and performs digital signal processing.

[1781] Step 2:

[1782] The server preprocesses the captured audio data. The input is digital audio data, and Librosa is used for noise reduction and normalization. The output is the preprocessed audio data. Specifically, a noise filtering algorithm is applied to equalize the volume levels.

[1783] Step 3:

[1784] The server inputs pre-processed audio data into a generative AI model. The input is pre-processed audio data, which is analyzed by the generative AI model (using TensorFlow or PyTorch). The output is data for conversion into sign language or Braille. Specifically, a speech recognition algorithm analyzes the audio data and generates corresponding sign language or Braille data.

[1785] Step 4:

[1786] The server uses an emotion engine to recognize the user's emotions from audio data. The input is pre-processed audio data, and emotions are identified using OpenCV or the Emotion API. The output is the recognized emotion data. Specifically, it analyzes the tone and intonation of the voice to identify emotion categories (e.g., surprise, joy, sadness).

[1787] Step 5:

[1788] The server generates information based on recognized emotions. Input consists of emotion data and sign language or Braille data, which are combined to generate information for the user. Output is sign language or Braille information corresponding to the emotion. Specifically, it adjusts the content of the sign language or Braille based on the emotion data and sets points to emphasize.

[1789] Step 6:

[1790] The device provides users with generated sign language or Braille information. Input is sign language or Braille information corresponding to emotions, and output is information displayed on a screen or Braille display. Specifically, in the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1791] (Application Example 2)

[1792] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1793] When people with visual or hearing impairments enjoy watching sports, they often have difficulty understanding live commentary in real time. Furthermore, because information is not provided in a way that responds to the user's emotions, there is a challenge in highlighting information that reflects their changing feelings. As a result, people with visual or hearing impairments are unable to fully enjoy watching sports.

[1794] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1795] In this invention, the server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data using a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and converting the information with emphasis into sign language or Braille. This makes it possible for visually and hearing impaired people to enjoy watching sports in real time. Furthermore, because information is emphasized according to the user's emotions, a more immersive viewing experience can be provided.

[1796] "Live broadcast audio data" refers to audio information obtained from live broadcasts of sporting events and other live events.

[1797] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[1798] Sign language is a visual gesture language used by visually impaired people to understand auditory information.

[1799] Braille is a pattern of raised dots used by visually impaired people to understand written information through touch.

[1800] An "emotion engine" is a technology that recognizes a user's emotions in real time and generates information that corresponds to those emotions.

[1801] "User emotions" refers to the emotional states, such as surprise, joy, and sadness, that users experience while watching sports.

[1802] "Highlighting information" means displaying specific information in a way that makes it stand out, depending on the user's emotions.

[1803] The system for carrying out this invention is configured as follows: The server includes means for acquiring live broadcast audio data, means for analyzing the live broadcast audio data with a generating AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and converting the information with emphasis into sign language or Braille.

[1804] Program Processing Description

[1805] The server uses the smartphone's microphone to acquire live broadcast audio data. The acquired audio data is converted to text using the speech_recognition library. Next, a generative AI model is used to convert the text data into sign language or Braille. In this process, TensorFlow is used to execute the sign language and Braille conversion model.

[1806] Furthermore, the emotion engine uses the emotion_recognition library to recognize the user's emotions in real time. Based on the recognized emotions, the generative AI emphasizes specific information and converts it into sign language or Braille. This makes it possible to provide information that is tailored to the user's emotions.

[1807] Specific example

[1808] For example, if a user is watching a live broadcast of a sporting event, the commentary audio is captured through the smartphone's microphone. The audio data is converted to text in real time, and that text can then be converted into sign language or Braille. Simultaneously, an emotion engine recognizes the user's emotions, and specific scenes or information are highlighted according to those emotions, such as surprise or joy.

[1809] Example of a prompt

[1810] "This application converts live commentary from sports events into sign language and Braille in real time, providing information to help visually and hearing-impaired individuals enjoy watching sports. Furthermore, it uses an emotion engine to recognize the user's emotions and highlights information corresponding to those emotions."

[1811] In this way, people with visual or hearing impairments can enjoy watching sports in real time, and information is emphasized according to the user's emotions, providing a more immersive viewing experience.

[1812] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[1813] Step 1:

[1814] The server uses the smartphone's microphone to acquire live audio data. The input is audio data from the smartphone's microphone, and the output is an audio data file. This audio data is used in subsequent processing steps.

[1815] Step 2:

[1816] The server converts the acquired audio data into text using the speech_recognition library. The input is the audio data acquired in step 1, and the output is text data. This conversion allows the audio information to be treated as text information.

[1817] Step 3:

[1818] The server uses a generative AI model to convert text data into sign language and Braille. The input is the text data obtained in step 2, and the output is sign language data and Braille data. This conversion uses a sign language conversion model and a Braille conversion model based on TensorFlow.

[1819] Step 4:

[1820] The server uses the emotion_recognition library to recognize the user's emotions in real time. The input is the text data obtained in step 2, and the output is the user's emotion data. This emotion data is used in subsequent information enhancement processing.

[1821] Step 5:

[1822] The server converts specific information into sign language or Braille, emphasizing it based on recognized emotion data. The input is the sign language and Braille data obtained in step 3, as well as the emotion data obtained in step 4, and the output is the emphasized sign language and Braille data. This process enables the provision of information tailored to the user's emotions.

[1823] Step 6:

[1824] The server outputs highlighted sign language and Braille data to the user. The input is the highlighted sign language and Braille data obtained in step 5, and the output is the sign language and Braille information displayed on the user's device. This allows visually and hearing impaired individuals to enjoy watching sports in real time.

[1825] (Example 3)

[1826] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1827] When users with visual or hearing impairments enjoy watching sports, there is a challenge in providing them with real-time information that responds to their emotions. In particular, the lack of information that takes users' emotions into consideration limits their experience, which is a problem.

[1828] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1829] In this invention, the server includes means for acquiring user emotion data, means for analyzing said emotion data, means for generating sign language or Braille data based on the analyzed emotion data, and means for outputting said sign language or Braille data. This makes it possible to provide real-time information in response to the user's emotions.

[1830] "User emotion data" refers to data that indicates the user's emotional state, obtained from the user's facial expressions, voice, and other sources.

[1831] "Means of acquisition" refers to devices and software used to collect user emotional data.

[1832] "Means of analysis" refers to devices or software used to analyze acquired emotional data and identify the user's emotional state.

[1833] "Generating means" refers to devices and software used to generate sign language or Braille data based on analyzed emotional data.

[1834] "Output methods" refer to devices and software used to provide users with generated sign language or Braille data.

[1835] "Sign language data" refers to data that contains information expressed in sign language.

[1836] "Braille data" refers to data that contains information expressed in Braille.

[1837] "Real-time" refers to data acquisition, analysis, generation, and output occurring without delay.

[1838] "Visually impaired or hearing impaired" refers to people who have a visual or hearing impairment.

[1839] Modes for carrying out the invention

[1840] This invention is a system that provides real-time information tailored to the user's emotions, enabling users with visual or hearing impairments to enjoy watching sports. Specific embodiments of this system are described below.

[1841] System Configuration

[1842] 1. Obtaining user sentiment data

[1843] The user inputs their facial expressions and voice through the device's camera and microphone. The device transmits this data to the server in real time. The hardware used includes a camera and microphone.

[1844] 2. Analysis of emotional data

[1845] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotions the user is feeling. For example, if the user has a sad expression, the emotion engine recognizes the emotion as "sadness." The software used includes the emotion engine.

[1846] 3. Generation of sign language and braille data

[1847] The server generates sign language and Braille data using a generative AI model based on emotional data obtained from the emotion engine. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if the user expresses sadness, the generative AI model will generate sign language and Braille data that emphasizes the sad scene.

[1848] 4. Data Output

[1849] The server transmits the generated sign language data to the terminal's display and the braille data to the braille display. The terminal displays the sign language on its screen and physically forms braille on the braille display. The user can receive information by looking at the display or touching the braille display. The hardware used includes a display and a braille display.

[1850] Specific example

[1851] As a concrete example, consider a scenario where a user is watching a sports event. If the user expresses sadness, the emotion engine recognizes this emotion, and the server uses a generative AI model based on this information to generate sign language or Braille data that emphasizes the sad scene. The generated data is then provided to the user through a display or Braille display.

[1852] Example of a prompt

[1853] "When a user expresses sadness while watching a sporting event, please recognize that emotion and generate sign language or Braille data that emphas...

Claims

1. A means of obtaining audio information from live broadcasts of sports events, A means for estimating the emotional state of a user watching the live broadcast using an emotion engine based on at least one of the user's facial expressions, tone of voice, and choice of words, means for generating a first prompt statement that instructs the conversion of the acquired audio information into text information, A means for generating text information corresponding to the acquired voice information using the generated first prompt sentence and the generating AI model, Means for generating a second prompt sentence that instructs the conversion of the generated text information into sign language data and braille data, and the processing to emphasize exciting scenes according to the estimated emotional state of the user, A means for generating sign language data and braille data that correspond to the generated text information and emphasize the exciting scene according to the estimated emotional state of the user, using the generated second prompt sentence and the generating AI model, A means for visually displaying the generated sign language data on a display using sign language animation software, means for outputting the generated braille data to a braille display, Includes, A system comprising the display and the braille display, which are devices for providing the sign language data and the braille data to the user.

2. A means for generating statistical information about important highlight moments and match content based on the generated text information, A means of outputting the generated highlight scenes and statistical information, The system according to claim 1, further comprising:

Images