System

Abstract

To provide a system which enables a visually impaired or hearing-impaired person to sufficiently receive information of live coverage when watching sports.SOLUTION: A system includes: means for acquiring audio information of live coverage in watching sports; means for generating a first prompt sentence instructing translation of the acquired audio information into text information; means for generating the text information corresponding to the acquired audio information using the generated first prompt sentence and a generative AI model; means for generating a second prompt sentence instructing a translation of the generated text information into sign language data or braille data; means for generating the sign language data or the braille data corresponding to the generated text information using the generated second prompt sentence and the generative AI model; means for visually displaying the generated sign language data using sign language animation software; and means for outputting the generated braille data onto a braille display.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When visually or hearing impaired people watch sports, they face the challenge of not being able to fully enjoy live broadcast information. [Means for solving the problem]

[0005] To solve this problem, we provide a system that takes in live broadcast audio data, analyzes it with a generation AI, and converts it into sign language or Braille in real time. Furthermore, the generation AI generates information based on the live broadcast audio data to help visually and hearing impaired people enjoy watching sports, and outputs that information in real time. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a microphone and audio input device are provided to capture live audio data from a sports broadcast. This audio input device can capture live audio data in real time. The captured audio data is then sent to the AI ​​generation system.

[0029] "Example 2"

[0030] The generative AI analyzes the captured voice data and converts it into sign language or Braille in real time.

[0031] Specifically, it generates sign language and Braille data based on information obtained from audio data. This generation AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[0032] "Example 3"

[0033] The generated sign language or Braille data is provided to the user through an output device. Examples of output devices include a display for displaying sign language and a Braille display for outputting Braille. A specific example is a Braille display, which physically generates Braille based on the generated Braille data, allowing visually impaired people to read it by touch.

[0034] The processing flow of each embodiment will be described below.

[0035] "Example 1"

[0036] Step 1: Provide a microphone or audio input device that captures live broadcast audio data of a sports event. This audio input device can capture live broadcast audio data in real time.

[0037] Step 2: The captured voice data is sent to the generation AI, which is trained using AI techniques such as deep learning to achieve highly accurate conversion.

[0038] "Example 2"

[0039] Step 1: The generation AI analyzes the captured voice data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the voice data.

[0040] Step 2: The generated sign language or Braille data is provided to the user through an output device, such as a display for displaying sign language or a Braille display for outputting Braille.

[0041] "Example 3"

[0042] Step 1: As a concrete example, a Braille display physically forms Braille characters based on the generated Braille data, allowing visually impaired people to read them tactilely.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] People with visual or hearing impairments have difficulty enjoying watching sports. In particular, they have difficulty understanding audio information from live broadcasts in real time, which significantly reduces the enjoyment of watching sports.

[0046] The specification processing by the specification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for importing live broadcast audio data, a means for transmitting the live broadcast audio data to the generative AI model, and a means for the generative AI model to analyze the audio data and convert it into text. This enables people with visual or hearing impairments to enjoy watching sports in real time.

[0047] "Live broadcast audio data" is audio information provided in real time by a commentator at an event such as a sports game.

[0048] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates a specific output.

[0049] A "means" is a method or device used to achieve a particular purpose.

[0050] "Convert to text" is the process of analyzing audio data and converting it into text information.

[0051] "Output" refers to providing the processed data in a display, audio, or other format for presentation to a user.

[0052] MODE FOR CARRYING OUT THE INVENTION

[0053] This invention is a system that captures live audio data from sports broadcasts in real time and sends that data to a generative AI model. Specifically, the system acquires audio data using a microphone or audio input device, and sends the data to a generative AI model, where it performs various data processing and calculations.

[0054] Hardware and software used

[0055] Hardware: Microphone, audio input device

[0056] Software: Generative AI models (e.g., OpenAI's GPT-4)

[0057] System Overview

[0058] 1. Audio data capture:

[0059] A user broadcasts a live sports event.

[0060] A microphone or audio input device connected to the terminal captures audio data from the live broadcast in real time.

[0061] 2. Sending audio data:

[0062] The captured voice data is transmitted from the terminal to the server.

[0063] The server sends the received voice data to the generative AI model.

[0064] 3. Data analysis and processing:

[0065] The generative AI model analyzes the received voice data and performs the necessary data processing and calculations.

[0066] For example, it converts audio data into text, extracts important events, and generates real-time highlights.

[0067] 4. Result output:

[0068] The server receives the results from the generative AI model and provides feedback to the user.

[0069] Specifically, the generated text is sent to the user's terminal, and the terminal displays the text.

[0070] Specific examples

[0071] Example 1:

[0072] A user is broadcasting a live soccer match.

[0073] The microphone picks up the sound of "Goal!"

[0074] The terminal transmits this audio data to the server.

[0075] The server sends the voice data to the generative AI model, which then generates the text "A goal has been scored!"

[0076] Example 2:

[0077] A user is broadcasting a basketball game.

[0078] The microphone picks up the sound of "Three-point shot made!"

[0079] The terminal transmits this audio data to the server.

[0080] The server sends the voice data to the generative AI model, which then generates the text "The three-point shot was successful!"

[0081] Prompt Sentence Examples

[0082] Prompt 1:

[0083] "Please analyze this audio data and extract the key events. Audio data: 'Goal!'"

[0084] Prompt Statement 2:

[0085] "Please convert this audio data to text. Audio data: 'Three-pointer made!'"

[0086] The above is an embodiment of the present invention.

[0087] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0088] Step 1:

[0089] A user is watching a sports game and broadcasting live commentary. A microphone and audio input device connected to the device captures the audio data of the live broadcast in real time. Specifically, when the user shouts "Goal!", the microphone captures the audio and converts it into digital data. The input is the user's voice, and the output is digital audio data.

[0090] Step 2:

[0091] The terminal sends the captured voice data to the server. Specifically, the terminal compresses the voice data and sends it to the server using the TCP / IP protocol. The input is digital voice data, and the output is the voice data sent to the server.

[0092] Step 3:

[0093] The server sends the received voice data to the generative AI model. Specifically, the server sends the voice data to the API of the generative AI model and waits for the analysis results. The input is the voice data sent to the server, and the output is the voice data sent to the generative AI model.

[0094] Step 4:

[0095] The generative AI model analyzes the voice data and performs the necessary data processing and calculations. Specifically, the generative AI model converts the voice data into text and generates the text "A goal has been scored!" The input is the voice data sent to the generative AI model, and the output is the generated text.

[0096] Step 5:

[0097] The server receives the results from the generative AI model and provides feedback to the user. Specifically, the server sends the generated text to the user's device, which then displays the text. The input is the generated text, and the output is the text displayed on the user's device.

[0098] (Application example 1)

[0099] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] In the past, when watching sports, it was difficult for people with visual or hearing impairments to understand the progress of the game or important highlights in real time. Furthermore, even for general spectators, there were limited means to obtain important highlights and statistical information about the game in real time. This limited the enjoyment of watching the game.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0102] In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information related to the game content based on the converted commentary content, and means for outputting the generated highlights and statistical information. This enables visually or hearing impaired people to obtain information in real time to enjoy watching sports, and also enables general spectators to obtain important highlights of the game and statistical information related to the game content in real time.

[0103] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports or events.

[0104] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate specific outputs.

[0105] "Sign language" is a language that uses hand and finger movements to communicate with people who are deaf or hard of hearing.

[0106] Braille is a tactile representation of letters made up of raised dots that is used by the visually impaired to read.

[0107] "Means for converting to text" refers to technology that analyzes voice data and converts it into corresponding text information.

[0108] "Highlights" refers to important moments or events in sports or events.

[0109] "Statistical Information" refers to numerical data and analytical results related to a match or event.

[0110] "Output means" refers to technology such as a display device or audio device for providing the generated information to the user.

[0111] A system for implementing this invention includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the textual commentary content, and means for outputting the generated highlights and statistical information.

[0112] Hardware and software used

[0113] Hardware: Smartphone (with built-in microphone)

[0114] Software: Python, SpeechRecognition library (voice recognition), OpenAI API (generative AI)

[0115] Processing flow

[0116] 1. Audio data capture:

[0117] The smartphone microphone is used to capture live audio data in real time. The user launches the smartphone application and starts voice input.

[0118] 2. Audio to text conversion:

[0119] The captured voice data is converted to text using the SpeechRecognition library. The server sends the voice data to the Google® Speech Recognition API and retrieves the corresponding text data.

[0120] 3. Generative AI analysis:

[0121] The transcribed commentary is sent to a generator AI using the OpenAI API, which analyzes the input text data and generates key highlights and statistics.

[0122] 4. Output of information:

[0123] The generated highlights and statistics are displayed in real time on the user's smartphone, allowing the user to keep track of key moments and statistics of the match in real time.

[0124] Specific examples

[0125] When a user is watching a soccer match, they launch a smartphone app and capture the audio from the live broadcast. The app converts the audio into text and displays important events such as "Goal!" and "Foul!" in real time. It also automatically generates and displays highlights and statistics from the match (e.g., number of shots, ball possession).

[0126] Prompt Sentence Examples

[0127] Generate highlights of the following sports plays:

[0128] "30 minutes into the first half, Team A scored a goal. Team B's defense collapsed. Team A continued to attack in the second half."

[0129] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0130] Step 1:

[0131] A user launches a smartphone application to capture the audio of a live broadcast. The user uses the smartphone's microphone to collect audio data in real time. The input is the audio data of the live broadcast, and the output is a stream of audio data.

[0132] Step 2:

[0133] The device converts the captured audio data into text using the SpeechRecognition library. The device sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data. The input is a stream of audio data, and the output is text data.

[0134] Step 3:

[0135] The server sends the textual commentary to the generation AI using the OpenAI API. The server analyzes the input text data and generates important highlights and statistics. The input is text data, and the output is the generated highlights and statistics.

[0136] Step 4:

[0137] The server sends the generated highlights and statistical information to the user's smartphone in real time, where the user can view the information on the smartphone screen. The input is the generated highlights and statistical information, and the output is the information displayed on the smartphone screen.

[0138] Step 5:

[0139] Users can understand key moments and statistical information of a match in real time through a smartphone application. Specifically, users check the information on the screen and understand the progress of the match. The input is the information displayed on the smartphone screen, and the output is the user's understanding.

[0140] Example 2

[0141] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] People with visual or hearing impairments have difficulty understanding audio information in real time. In particular, the accuracy and speed of converting audio information into sign language or Braille is insufficient, which can delay the transmission of information or lead to misunderstandings. This makes it difficult for people with visual or hearing impairments to receive information accurately and quickly.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0144] In this invention, the server includes a means for capturing voice data, a means for converting the voice data into text data using voice recognition software, and a means for inputting the text data into a generative AI model to generate sign language data or Braille data, thereby enabling visually or hearing impaired people to receive voice information in sign language or Braille in real time.

[0145] "Audio data" refers to data in which audio is recorded in digital format.

[0146] "Speech recognition software" is software that analyzes voice data and converts it into text data.

[0147] "Text data" refers to data in which character information is recorded in digital format.

[0148] A "generative AI model" is an artificial intelligence model that is trained using deep learning technology and generates sign language data or Braille data based on text data.

[0149] "Sign language data" refers to data that records sign language movements in digital format.

[0150] "Braille data" refers to data that records Braille information in digital format.

[0151] "Sign language animation software" is software that visually displays sign language movements based on sign language data.

[0152] A "Braille display" is a device that physically displays Braille based on Braille data.

[0153] This invention is a system that enables people with visual or hearing impairments to receive audio information in real time as sign language or Braille. A specific embodiment of this system will be described below.

[0154] System configuration

[0155] Server Configuration

[0156] The server includes a means for capturing voice data, a means for converting the voice data into text data using voice recognition software, and a means for converting the text data into sign language data or Braille data using a generative AI model. The voice recognition software uses a general voice recognition API (e.g., Google Cloud Speech-to-Text API). The generative AI model is a model trained using deep learning technology (e.g., a model using TENSORFLOW (registered trademark) or PyTorch).

[0157] Device configuration

[0158] The terminal includes a means for visually displaying the sign language data sent from the server using sign language animation software (e.g., SignAll), and a means for outputting Braille data to a Braille display (e.g., Focus Blue). The terminal also includes a microphone for the user to input voice data.

[0159] User operations

[0160] The user inputs voice data by speaking into the device's microphone. For example, they might say "hello." The device then sends this voice data to the server. The server converts the voice data into text data and generates sign language or Braille data using a generative AI model. The generated data is then sent to the device and displayed on the display as sign language animation or output to a Braille display.

[0161] Specific examples

[0162] Example 1: Conversion to sign language

[0163] When a user says "Good morning," the device sends the voice data to the server. The server converts the voice data into text and inputs the text information "Good morning" into the generative AI model. The generative AI model generates sign language data corresponding to "Good morning" and creates a sign language animation using sign language animation software. The device displays this sign language animation on the display, allowing the user to visually confirm the sign language for "Good morning."

[0164] Example 2: Conversion to Braille

[0165] When a user says "Thank you," the device sends the voice data to the server. The server converts the voice data into text and inputs the text information "Thank you" into the generative AI model. The generative AI model generates Braille data corresponding to "Thank you" and sends it to the Braille display. The Braille display connected to the device displays "Thank you" in Braille, allowing the user to read the Braille with their sense of touch.

[0166] Prompt Sentence Examples

[0167] "If a user says 'Good morning,' how would your generative AI model convert that into sign language or braille?"

[0168] By inputting this prompt into a generative AI model, a detailed explanation of the specific conversion process can be obtained.

[0169] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0170] Step 1:

[0171] The user inputs voice data.

[0172] The user speaks into the device's microphone. For example, they say "hello." The device captures this speech as digital audio data. The input is the user's voice, and the output is digital audio data.

[0173] Step 2:

[0174] The terminal transmits the voice data to the server.

[0175] The device transmits the captured audio data over the Internet to a server, where it is encoded in an appropriate format (e.g., WAV or MP3). The input is the digital audio data, and the output is the audio data transmitted to the server.

[0176] Step 3:

[0177] The server converts the voice data into text data.

[0178] The server converts the received voice data into text data using voice recognition software (for example, a general voice recognition API). For example, voice data saying "Hello" is converted into text data saying "Hello." The input is voice data, and the output is text data.

[0179] Step 4:

[0180] The server inputs the text data into the generative AI model.

[0181] The server inputs the converted text data into a generative AI model, which is trained using deep learning techniques. The input is text data, and the output is sign language data or Braille data.

[0182] Step 5:

[0183] A generative AI model generates sign language and Braille data.

[0184] A generative AI model generates sign language data or Braille data based on input text data. For example, the text data "Hello" is converted into sign language animation data or Braille data. The input is text data, and the output is sign language data or Braille data.

[0185] Step 6:

[0186] The server transmits the generated data to the terminal.

[0187] The server sends the generated sign language or braille data to the device. The data is encoded in an appropriate format (e.g., JSON or XML). The input is the sign language or braille data, and the output is the data sent to the device.

[0188] Step 7:

[0189] The device displays sign language animations and outputs braille data to a braille display.

[0190] The terminal displays the received sign language data on a display using sign language animation software (e.g., sign language animation software). In addition, the Braille data is output to a Braille display (e.g., a Braille display) connected to the terminal. The user can visually check the sign language animation displayed on the display and tactilely read the Braille displayed on the Braille display. The input is sign language data or Braille data, and the output is sign language animation or Braille.

[0191] (Application example 2)

[0192] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] People with visual or hearing impairments have difficulty understanding voice instructions and warnings in real time during security work, which can prevent them from working efficiently and lead to a decline in the quality of security.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting audio data into text, means for converting the text into sign language and Braille, and means for providing the sign language and Braille to visually impaired and hearing impaired persons. This enables visually impaired and hearing impaired persons to understand audio instructions and warnings in real time and perform security operations efficiently.

[0195] "Live broadcast audio data" refers to audio information about events or situations occurring in real time.

[0196] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[0197] "Sign language" refers to a means of communicating linguistic information using visual gestures and movements.

[0198] "Braille" refers to a writing system that uses raised dots to enable visually impaired people to read written information using their sense of touch.

[0199] "Audio data" refers to information that records sound in digital form.

[0200] "Text" refers to textual information.

[0201] "Blind people" refers to people who are visually impaired.

[0202] "Deaf" refers to people who have hearing impairments.

[0203] "Security work" refers to work to protect the safety of facilities and people.

[0204] "Real-time" refers to processing and transmission occurring immediately without delay.

[0205] A system for carrying out the present invention converts voice data into sign language or Braille in real time and provides the converted data to visually impaired or hearing impaired people. A specific embodiment of the system will be described below.

[0206] System configuration

[0207] Hardware

[0208] Smartphone: Equipped with a microphone for capturing voice data and a display for showing the conversion results.

[0209] Server: Provides the computing resources for analyzing and converting audio data.

[0210] software

[0211] Speech Recognition Library: Use the speech_recognition library to convert audio data to text.

[0212] Sign Language Conversion Library: Use the hand_gesture_recognition library to convert text to sign language.

[0213] Braille conversion library: Use the braille_converter library to convert text to braille.

[0214] Generative AI models: Use deep learning models to analyze audio data and convert it into sign language or Braille.

[0215] System Operation

[0216] Acquiring audio data

[0217] The user inputs voice data using the smartphone's microphone. For example, a security guard may input "There is a suspicious person here."

[0218] Analysis of audio data

[0219] The server converts the captured audio data into text using the speech_recognition library, which serves as the basis for subsequent sign language and braille conversion.

[0220] Conversion to sign language and braille

[0221] The server uses generative AI models to convert text to sign language and braille using the hand_gesture_recognition library for sign language conversion and the braille_converter library for braille conversion.

[0222] Providing results

[0223] The converted sign language and Braille are then displayed on the smartphone's display, allowing visually and hearing impaired people to understand voice instructions and warnings in real time.

[0224] Specific examples

[0225] For example, if a security guard says "There's a suspicious person here," the system converts this speech into text and then into sign language and Braille, which are then displayed on the smartphone screen, making the information accessible to security guards who are visually or hearing impaired.

[0226] Prompt Sentence Examples

[0227] Enter the voice data. Example: "There is a suspicious person."

[0228] In this way, visually and hearing impaired people can understand voice instructions and warnings in real time and perform security tasks efficiently.

[0229] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0230] Step 1:

[0231] The user inputs voice data using the smartphone's microphone, which is temporarily stored in the smartphone's internal memory.

[0232] Step 2:

[0233] The smartphone sends the saved voice data to the server. The server receives the voice data and converts it into text using the speech_recognition library. The input is voice data and the output is text data.

[0234] Step 3:

[0235] The server analyzes the text data using a generative AI model. This analysis converts the text data into sign language and Braille. The input is text data, and the output is sign language and Braille data.

[0236] Step 4:

[0237] The server converts the sign language data into sign language using the hand_gesture_recognition library. The input is text data and the output is sign language data.

[0238] Step 5:

[0239] The server converts the braille data into braille using the braille_converter library. The input is text data and the output is braille data.

[0240] Step 6:

[0241] The server then sends the converted sign language and Braille data to the smartphone, which then displays the received data on its display, enabling visually and hearing impaired people to understand voice instructions and warnings in real time.

[0242] Example 3

[0243] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] The purpose of this system is to solve the problem that users with visual or hearing impairments have difficulty converting text data into sign language or Braille and using information in real time. In particular, there is a need for a system that can quickly and accurately convert text data entered by users into sign language or Braille and provide it through an appropriate output device.

[0245] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0246] In this invention, the server includes a means for a user to input text data, a means for the server to receive the input data and send a prompt to the generative AI model, a means for the generative AI model to generate sign language or Braille data, a means for the server to send the generated data to an output device, and a means for the terminal to provide the sign language or Braille to the user through the output device, thereby enabling visually or hearing impaired users to quickly and accurately convert input text data into sign language or Braille and use information in real time.

[0247] "User" refers to a person who uses the system to input text data and receive the information converted into sign language or Braille.

[0248] "Server" refers to a computer system whose role is to receive input data from a user, send prompts to a generative AI model, and send the generated sign language or Braille data to an output device.

[0249] A "generative AI model" refers to an artificial intelligence model that generates sign language or Braille data based on text data from the user.

[0250] A "prompt sentence" refers to a sentence that instructs a generative AI model to convert text data into sign language or Braille.

[0251] An "output device" is a device for providing the generated sign language or Braille data to a user, and includes a sign language display, a Braille display, and the like.

[0252] "Sign language display" refers to a display device for visually displaying generated sign language data.

[0253] "Braille display" refers to a device that physically renders generated Braille data so that it can be read by the visually impaired through touch.

[0254] "Text data" refers to textual information that a user inputs into a system.

[0255] "Real-time" means that text data entered by the user is instantly converted into sign language or Braille and provided almost simultaneously through an output device.

[0256] MODE FOR CARRYING OUT THE INVENTION

[0257] This invention is a system that enables visually or hearing impaired users to convert text data into sign language or Braille and access information in real time. A specific embodiment of this system will be described below.

[0258] System configuration

[0259] The system consists of the following major components:

[0260] 1. User terminal: A device that allows a user to input text data. Examples include PCs, smartphones, and tablets.

[0261] 2. Server: Receives input data from the user, sends prompts to the generative AI model, and sends the generated sign language or Braille data to the output device. Use a high-performance server (e.g., a server equipped with an NVIDIA GPU).

[0262] 3. Generative AI model: An artificial intelligence model for generating sign language or Braille data based on text data from users. It uses deep learning frameworks such as TensorFlow and PyTorch.

[0263] 4. Output device: A device that presents the generated sign language or Braille data to the user, including a sign language display or a Braille display.

[0264] System Operation

[0265] 1. The user enters text data.

[0266] A user inputs text data through the system's interface, for example, by typing "hello" using a web browser or a dedicated application.

[0267] 2. The server receives the input data and sends a prompt to the generative AI model.

[0268] The server receives the text data sent by the user. Based on the received data, it creates a prompt sentence and sends it to the generative AI model. For example, it sends the prompt sentence "Please convert the following text into sign language: Hello" to the generative AI model.

[0269] 3. The generative AI model generates sign language and Braille data.

[0270] The generative AI model analyzes the prompt received from the server and generates corresponding sign language or Braille data, such as sign language animation data or Braille sequence data.

[0271] 4. The server sends the generated data to an output device.

[0272] The server sends the sign language and Braille data received from the generative AI model to an output device, for example, sending sign language animation data to a sign language display and Braille data to a Braille display.

[0273] 5. The terminal provides sign language or Braille to the user through an output device.

[0274] The terminal receives the data sent from the server and provides it to the user through the output device. The sign language display shows the sign language animation, and the braille display physically generates braille. The user can see the sign language display visually and read the braille display tactilely.

[0275] Specific examples

[0276] For example, when the user inputs the text "Thank you," the specific operation is as follows.

[0277] 1. User: Enters "Thank you" into the input field in a web browser and clicks the send button.

[0278] 2. Server: Receives the text data "Thank you" from the user and sends the prompt sentence "Please convert the following text into sign language: Thank you" to the generative AI model.

[0279] 3. Generative AI model: Analyzes the prompt and generates sign language animation data corresponding to "thank you."

[0280] 4. Server: Sends the generated sign language animation data to the sign language display.

[0281] 5. Terminal: The sign language display shows the sign language animation, which the user can visually confirm.

[0282] In this way, the user can easily convert text into sign language or Braille and use it through an output device. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0283] Step 1:

[0284] The user inputs text data.

[0285] A user inputs text data using a web browser or a dedicated application. For example, the user inputs "Hello" and clicks the send button. The input data is sent from the user's device to the server.

[0286] Step 2:

[0287] The server receives the input data and sends prompts to the generative AI model.

[0288] The server receives text data sent by the user. Based on the received data, it creates a prompt sentence and sends it to the generative AI model. For example, the prompt sentence "Please convert the following text into sign language: Hello" is sent to the generative AI model. The input is the user's text data, and the output is the prompt sentence to the generative AI model.

[0289] Step 3:

[0290] A generative AI model generates sign language and Braille data.

[0291] The generative AI model analyzes the prompt received from the server and generates the corresponding sign language or Braille data. For example, it generates sign language animation data or Braille sequence data. The input is the prompt, and the output is the sign language or Braille data.

[0292] Step 4:

[0293] The server sends the generated data to an output device.

[0294] The server sends the sign language or Braille data received from the generative AI model to an output device. For example, it sends sign language animation data to a sign language display and Braille data to a Braille display. The input is the generated sign language or Braille data, and the output is the data sent to the output device.

[0295] Step 5:

[0296] The terminal provides sign language or Braille to the user through an output device.

[0297] The terminal receives data sent from the server and provides it to the user through the output device. The sign language display shows sign language animations, and the braille display physically produces braille. The user can see the sign language display visually and read the braille display tactilely. The input is sign language or braille data from the server, and the output is the provision of sign language or braille to the user.

[0298] (Application example 3)

[0299] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] People with visual or hearing impairments have difficulty receiving real-time information when watching sports or obtaining security information. In addition, there is a need for rapid communication of information in sign language or Braille in emergencies, but current systems are inadequate.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing security camera footage, analyzing it using a sign language recognition model, and recognizing the sign language, means for displaying the recognized sign language on the footage, and means for outputting security information on a Braille display. This enables people with visual or hearing impairments to watch sports and receive security information in real time.

[0302] "Live broadcast audio data" refers to data that provides audio information about the progress of sporting events and other live events.

[0303] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[0304] "Sign language" is a visual, gestural language used by deaf people to communicate.

[0305] "Braille" is a writing system using raised dots that is used by the visually impaired to read by touch.

[0306] A "sign language recognition model" is a machine learning model that analyzes sign language movements from camera footage and converts them into the corresponding language.

[0307] A "security camera" is a camera device for monitoring a specific area and acquiring video data.

[0308] A "Braille display" is a device that displays Braille characters using electronically controlled raised dots.

[0309] "Security information" is information relating to abnormalities or emergencies within the monitored area.

[0310] A system for implementing this invention is configured as follows: The server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing security camera footage, analyzing it using a sign language recognition model, and recognizing the sign language, means for displaying the recognized sign language on the footage, and means for outputting security information on a Braille display.

[0311] Hardware and software used

[0312] Hardware:

[0313] Smartphone camera

[0314] Braille displays (e.g. BrailleDisplay)

[0315] security cameras

[0316] software:

[0317] OpenCV: Acquiring and displaying camera images

[0318] HandSignRecognition: A sign language recognition model

[0319] BrailleDisplay: A library for controlling braille displays

[0320] Data processing and calculation

[0321] 1. Importing live broadcast audio data:

[0322] The server captures play-by-play audio data that provides audio updates on the progress of sporting events and other live events.

[0323] 2. Generative AI analysis:

[0324] The server analyzes the captured live broadcast audio data using a generation AI and converts it into sign language or Braille in real time.

[0325] 3. Sign language and Braille output:

[0326] The server outputs the generated sign language or Braille through a display or Braille display.

[0327] 4. Security camera capture:

[0328] The server captures the security camera footage and analyzes it using a sign language recognition model.

[0329] 5. Sign Language Recognition and Display:

[0330] The server recognizes the sign language in the video using a sign language recognition model and displays the recognized sign language on the video.

[0331] 6. Outputting security information to a braille display:

[0332] The server outputs security information to a braille display.

[0333] Specific examples

[0334] Sports viewing:

[0335] Users with visual or hearing impairments can receive information from live broadcasts of sporting events by converting audio data into sign language or Braille in real time through a display or Braille display.

[0336] Obtaining security information:

[0337] The security camera captures the video, recognizes the sign language using a sign language recognition model, and displays the recognized sign language on the video. Security information is also output to a Braille display, allowing visually impaired people to receive the information tactilely.

[0338] Prompt Sentence Examples

[0339] "Create a program that analyzes security camera footage, displays emergency information in sign language, and outputs security information on a braille display."

[0340] The above is an embodiment of the present invention.

[0341] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0342] Step 1:

[0343] The server captures play-by-play audio data that provides audio information about the progress of a sporting event or other live event. The input is the play-by-play audio data, and the output is the captured audio data, which is used for subsequent analysis.

[0344] Step 2:

[0345] The server analyzes the captured live broadcast audio data using a generation AI and converts it into sign language or Braille in real time. The input is the live broadcast audio data, and the output is sign language data and Braille data. The generation AI analyzes the audio data and generates the corresponding sign language or Braille.

[0346] Step 3:

[0347] The server outputs the generated sign language and braille data through a display or braille display. The input is sign language data and braille data, and the output is sign language displayed on the display and braille displayed on the braille display. This allows users to receive information even if they are visually or hearing impaired.

[0348] Step 4:

[0349] The server captures the security camera footage and analyzes it using a sign language recognition model. The input is the security camera footage and the output is the recognized sign language data. The sign language recognition model analyzes the footage and recognizes the sign language movements.

[0350] Step 5:

[0351] The server displays the recognized sign language on the video. The input is the recognized sign language data, and the output is the sign language displayed on the video. This allows the user to understand the security camera video in sign language.

[0352] Step 6:

[0353] The server outputs security information to a braille display. The input is security information, and the output is a braille display on the braille display. This allows the user to receive security information tactilely.

[0354] The above are the specific processing steps for carrying out the present invention.

[0355] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0356] "Example 1"

[0357] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes a user's emotions and generates information based on those emotions to help visually or hearing-impaired people enjoy watching sports more. Specifically, the system infers emotions from the user's facial expressions, tone of voice, and word choice, and analyzes live broadcast audio data based on those emotions. For example, if a user expresses joy, the system recognizes that emotion and converts it into sign language or Braille, emphasizing the more exciting scenes.

[0358] "Example 2"

[0359] The emotion engine also recognizes emotions in real time and generates information based on those emotions. This makes it possible to provide information tailored to the user's real-time emotions, helping visually and hearing impaired people enjoy watching sports. For example, if a user expresses surprise, the system recognizes that emotion, highlights the surprise, and converts it into sign language or Braille.

[0360] "Example 3"

[0361] Furthermore, the emotion engine generates information based on the user's emotions and outputs that information in real time. This makes it possible to instantly provide information tailored to the user's emotions so that visually and hearing impaired people can enjoy watching sports. Specifically, if a user expresses sadness, the system recognizes that emotion, highlights sad scenes, and converts them into sign language or Braille.

[0362] The processing flow of each embodiment will be described below.

[0363] "Example 1"

[0364] Step 1: Estimate the user's emotions from their facial expressions, tone of voice, choice of words, etc.

[0365] Step 2: Analyze the live broadcast audio data according to the estimated emotions.

[0366] Step 3: Based on the analysis results, highlight the more exciting scenes and convert them into sign language or Braille.

[0367] "Example 2"

[0368] Step 1: Recognize user emotions in real time.

[0369] Step 2: Analyze the live broadcast audio data according to the recognized emotions.

[0370] Step 3: Based on the analysis results, highlight surprising scenes and convert them into sign language or Braille.

[0371] "Example 3"

[0372] Step 1: Recognize user emotions in real time and generate information based on those emotions.

[0373] Step 2: Output the generated information in real time.

[0374] Step 3: Based on the output information, highlight the sad scenes and convert them into sign language or Braille.

[0375] Example 1

[0376] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0377] When viewing sports, visually or hearing impaired users face the challenge of having difficulty understanding live audio information in real time. Another problem is that the enjoyment of watching a sporting event is not fully conveyed because information cannot be provided in response to the user's emotions.

[0378] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for importing live broadcast audio data, a means for analyzing the live broadcast audio data using a generative AI model and recognizing the user's emotion, and a means for converting the live broadcast audio data into sign language or Braille in real time according to the user's emotion. This makes it possible to provide information in real time so that users with visual or hearing impairments can enjoy watching sports.

[0379] "Live broadcast audio data" refers to data that provides real-time audio information about the progress of sporting events and other live events.

[0380] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to analyze data and generate information for a specific purpose.

[0381] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, tone of voice, choice of words, etc. to estimate their emotional state.

[0382] "Means for converting into sign language or Braille" refers to software or hardware for converting audio data or text data into sign language video or Braille information.

[0383] "Means for outputting sign language or Braille" refers to an output device such as a display or Braille display that provides the generated sign language video or Braille information to the user.

[0384] An "exciting scene" is an important moment or play that creates particular excitement and emotion in watching sports.

[0385] This invention is a system that allows visually or hearing impaired users to enjoy watching sports. The system captures live broadcast audio data in real time, recognizes the user's emotions, and converts information corresponding to those emotions into sign language or Braille and provides it.

[0386] Hardware and software used

[0387] Hardware

[0388] Microphones: Installed in stadiums and event venues, they capture live audio data in real time.

[0389] Camera: Used to capture the user's facial expressions.

[0390] Terminal: A device that processes audio and video data and sends it to a server.

[0391] Braille display: An output device for providing braille information to a user.

[0392] software

[0393] Generative AI model: An algorithm that analyzes voice data and recognizes user emotions.

[0394] Emotion engine: Software that estimates emotions from a user's facial expressions, tone of voice, choice of words, etc.

[0395] Sign language translation software: Software for converting audio and text data into sign language video.

[0396] Braille conversion software: Software for converting audio and text data into Braille information.

[0397] Specific operation of the system

[0398] Importing audio data

[0399] The device uses high-sensitivity microphones installed in the stadium to capture live audio data in real time, for example, capturing the audio of the commentator shouting "Goal!"

[0400] Sending audio data

[0401] The device then transmits the captured audio data over the internet to a cloud-based generative AI model, where it is compressed and sent to a server using a secure communication protocol.

[0402] User emotion recognition

[0403] The server monitors the user's connection to the system via a camera and microphone. The emotion engine analyzes the user's facial expressions (e.g., smile or surprise) and tone of voice (e.g., excitement) in real time to estimate the user's emotions.

[0404] Emotion-based data analysis

[0405] If the server estimates that the user's emotion is joy, it will highlight exciting scenes from the live broadcast audio data, for example, by identifying and extracting the moments of goals and important plays.

[0406] Information generation and provision

[0407] The server converts the extracted exciting scenes into sign language and Braille. Specifically, it uses sign language translation software to generate sign language videos and Braille conversion software to display them on a Braille display. The generated information is sent to the user's device and displayed in real time.

[0408] Examples and prompts

[0409] Specific examples

[0410] If a user is watching a soccer match and shows a facial expression of joy when a goal is scored, the system will recognize that emotion and convert the commentary of the goal into sign language or Braille, highlighting it.

[0411] Prompt Sentence Examples

[0412] "Capture the audio of a live broadcast of a soccer game in real time, the moment a goal is scored, recognize the user's emotion of joy, and convert that scene into sign language or Braille."

[0413] In this way, the system provides information that responds to the user's emotions, allowing visually and hearing impaired people to enjoy watching sports more.

[0414] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0415] Step 1:

[0416] The device captures live audio data in real time using high-sensitivity microphones installed in the stadium. The input is the live audio data, and the output is the captured audio data. Specifically, the device uses a noise-canceling function to remove background noise.

[0417] Step 2:

[0418] The device transmits the captured voice data to the generative AI model on the cloud via the internet. The input is the captured voice data, and the output is the voice data sent to the generative AI model. Specifically, the voice data is compressed and sent to the server using a secure communication protocol.

[0419] Step 3:

[0420] The server monitors the user's connection to the system through a camera and microphone. The input is the user's facial expression data and tone of voice data, and the output is the data sent to the emotion engine. Specifically, the camera captures the user's facial expression, and the microphone collects the user's tone of voice.

[0421] Step 4:

[0422] The server uses an emotion engine to estimate the user's emotions from their facial expressions, tone of voice, and choice of words. The input is the user's facial expression data and tone of voice data, and the output is estimated emotion data. Specifically, the emotion engine analyzes the data in real time and estimates the user's emotions.

[0423] Step 5:

[0424] If the server estimates that the user's emotion is joy, it highlights exciting scenes in the live broadcast audio data. The input is the estimated emotion data and the live broadcast audio data, and the output is the data of the highlighted exciting scenes. Specifically, the server uses an algorithm to detect specific keywords (e.g., "goal" or "great play") in the live broadcast audio data.

[0425] Step 6:

[0426] The server converts the extracted exciting scenes into sign language or Braille. The input is the data of the highlighted exciting scenes, and the output is sign language video or Braille information. Specifically, the data is converted using sign language translation software or Braille conversion software.

[0427] Step 7:

[0428] The server sends the generated sign language video and Braille information to the user's device. The input is the sign language video and Braille information, and the output is the sign language video and Braille information displayed on the user's device. Specifically, a low-latency communication protocol is used to ensure real-time performance.

[0429] In this way, the system provides information that responds to the user's emotions, allowing visually and hearing impaired people to enjoy watching sports more.

[0430] (Application example 1)

[0431] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0432] When visually or hearing impaired users enjoy watching sports, simply converting live audio data into sign language or Braille in real time poses a challenge as it does not provide enough information that reflects the user's emotions.In addition, exciting scenes are not emphasized in a way that takes the user's emotions into consideration, making it difficult for visually or hearing impaired users to fully enjoy watching sports.

[0433] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0434] In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, and means for emphasizing exciting scenes according to the emotions. This makes it possible to provide information according to the user's emotions so that visually or hearing impaired people can enjoy watching sports more.

[0435] "Live broadcast audio data" refers to audio information that conveys the progress of a sporting event or other live event in real time.

[0436] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate specific outputs.

[0437] "Sign language" is a means of communicating linguistic information using visual gestures and movements.

[0438] "Braille" is a pattern of raised dots on paper or other surfaces that can be read by the visually impaired using their sense of touch.

[0439] "Means for recognizing user emotions" refers to technology that estimates emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0440] "Means for emphasizing exciting scenes" refers to a technique for selecting particularly exciting scenes in a sports game and emphasizing those scenes for the visually or hearing impaired.

[0441] A system for implementing this invention is configured as follows: First, a server is equipped with a microphone and an audio input device for capturing live broadcast audio data. This audio input device can capture live broadcast audio data of a sporting event in real time.

[0442] The captured voice data is then sent to a generative AI, which analyzes the voice data and converts it into sign language or Braille in real time, which is then output in a format that can be understood by people with visual or hearing impairments.

[0443] Furthermore, the system is equipped with an emotion recognition engine that estimates the user's emotions from their facial expressions, tone of voice, choice of words, etc. The emotion recognition engine acquires and analyzes the user's emotional data using a camera and microphone.

[0444] Once the user's emotions are recognized, the system will highlight exciting scenes accordingly. For example, if the user expresses joy, the system will recognize that emotion and highlight particularly exciting scenes. This highlighted information can then be converted into sign language or Braille and provided to the visually or hearing impaired.

[0445] The specific hardware used is a microphone, camera, and server. The software uses Python, the speech_recognition library, the transformers library, and the opencv library. The combination of this hardware and software enables voice data capture, emotion recognition, conversion to sign language and braille, and highlighting exciting scenes.

[0446] For example, if a user expresses joy while watching a sports game, the system will recognize that emotion and convert it into sign language or Braille, highlighting particularly exciting scenes, and provide this information to the user in real time.

[0447] Examples of prompts to input to a generative AI model include:

[0448] If the user is showing the emotion of joy, how can you highlight the exciting scene and translate it into sign language or braille?

[0449] This prompt can be used to ask the generative AI model how to convert specific sign language or braille.

[0450] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0451] Step 1:

[0452] The server captures live broadcast audio data in real time using a microphone.

[0453] Input: Live broadcast audio data

[0454] Output: Captured audio data

[0455] Specific operation: The server captures the audio signal from the microphone as digital data and performs preprocessing for voice recognition.

[0456] Step 2:

[0457] The server sends the captured voice data to the generation AI for analysis.

[0458] Input: Captured audio data

[0459] Output: Analyzed audio data

[0460] Specific operation: The server inputs the voice data into the generative AI model, which performs speech recognition and natural language processing, converting the voice data into text data.

[0461] Step 3:

[0462] The server converts the analyzed voice data into sign language or Braille in real time.

[0463] Input: Parsed audio data

[0464] Output: Sign language data, Braille data

[0465] How it works: The server uses a generative AI model to run an algorithm that converts text data into sign language or Braille. The sign language data is output in a video format, and the Braille data is output in a format suitable for a tactile display.

[0466] Step 4:

[0467] The server uses a camera to capture the user's facial expressions and sends them to an emotion recognition engine.

[0468] Input: User's facial expression data

[0469] Output: Recognized emotion data

[0470] Specific operation: The server acquires video data from the camera and inputs it into the emotion recognition engine, which analyzes facial expressions and estimates the user's emotions.

[0471] Step 5:

[0472] The server highlights exciting scenes based on the recognized emotion data.

[0473] Input: Recognized emotion data, sign language data, Braille data

[0474] Output: Enhanced sign language data, Enhanced Braille data

[0475] Specific operation: The server analyzes the emotion data and selects and highlights particularly exciting scenes if the user expresses joy, thereby adjusting the sign language data and Braille data according to the user's emotion.

[0476] Step 6:

[0477] The server provides enhanced sign language and Braille data to the user.

[0478] Input: Enhanced sign language data, enhanced braille data

[0479] Output: Sign language data and Braille data provided to the user

[0480] Specific operation: The server sends the enhanced sign language data to the user in video format and the Braille data in a format suitable for tactile displays. The user receives this data in real time and can enjoy watching sports.

[0481] Example 2

[0482] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0483] When users with visual or hearing impairments enjoy watching sports, they face the challenge of having difficulty understanding the audio information of live broadcasts in real time.In addition, they face the problem of not being able to share the sense of realism and excitement of the event because the information provided is not tailored to the user's emotions.

[0484] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0485] In this invention, the server includes means for capturing live broadcast audio data, means for preprocessing the live broadcast audio data, means for analyzing the preprocessed audio data using a generative AI model and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing a user's emotion from the audio data, and means for generating information based on the recognized emotion, thereby enabling users with visual or hearing impairments to enjoy watching sports in real time.

[0486] "Live broadcast audio data" is audio information provided in real time during live broadcasts of sports, events, etc.

[0487] "Preprocessing" refers to processes such as noise removal and normalization that are carried out to convert audio data into a format that is easier to analyze.

[0488] A "generative AI model" is an artificial intelligence model trained using deep learning techniques to analyze audio data and convert it into sign language or Braille.

[0489] "Sign language" is a visual, gestural language used by the blind to understand spoken information.

[0490] "Braille" is a writing system using raised dots that is used by visually impaired people to understand written information by touch.

[0491] "Emotion recognition" is an analytical technique for identifying a user's emotions from voice data.

[0492] "Information generation" is the process of creating information to provide to the user based on the analyzed voice data and recognized emotions.

[0493] "Output means" refers to a device or interface for providing the generated sign language or Braille information to the user.

[0494] The present invention is a system that enables visually or hearing impaired users to enjoy watching sports in real time. A specific embodiment of this system will be described below.

[0495] System configuration

[0496] Hardware

[0497] The server uses a microphone (e.g., a standard microphone) to capture voice data, has a high-performance processor and memory for processing and analyzing the voice data, and has a display or Braille display for outputting information in sign language or Braille.

[0498] software

[0499] The server uses the Python library Librosa to preprocess the audio data. The generative AI model is built using TensorFlow and PyTorch to analyze the audio data and convert it into sign language or Braille. OpenCV and the Emotion API are used for emotion recognition.

[0500] System Operation

[0501] The server captures live audio data from a microphone in real time. The captured audio data undergoes pre-processing such as noise reduction and normalization using Librosa. The pre-processed audio data is then fed into a generative AI model and converted into sign language or Braille.

[0502] At the same time, the server uses an emotion engine to recognize the user's emotions from the voice data in real time. Based on the recognized emotions, information tailored to the user's emotions is generated. For example, if the user expresses surprise, the system recognizes that emotion and converts it into sign language or Braille, emphasizing the surprise.

[0503] The generated sign language or Braille information is provided to the user via the device. In the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[0504] Specific examples

[0505] When a user exclaims "Amazing!" while watching a sports game, the server captures the speech and inputs it into a generative AI model. The generative AI model analyzes the speech and generates data that converts the "Amazing!" into sign language or Braille. At the same time, the emotion engine recognizes the user's emotion of surprise and generates information that emphasizes the surprising scene according to that emotion.

[0506] Prompt Sentence Examples

[0507] "Describe a system that converts a user's voice of surprise while watching a sports game into sign language or Braille in real time, highlighting the surprising moment and providing information to the visually or hearing impaired."

[0508] This system allows users with visual or hearing impairments to enjoy watching sports in real time.

[0509] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0510] Step 1:

[0511] The server captures live audio data in real time through a microphone. The input is analog audio data, which is converted to digital format and stored in temporary storage. Specifically, the audio input device captures the audio and performs digital signal processing.

[0512] Step 2:

[0513] The server performs pre-processing on the captured audio data. The input is digital audio data, which is denoised and normalised using Librosa. The output is pre-processed audio data. Specifically, a noise filtering algorithm is applied and the volume level is equalised.

[0514] Step 3:

[0515] The server inputs preprocessed speech data into the generative AI model. The input is preprocessed speech data, which is analyzed by the generative AI model (using TensorFlow or PyTorch). The output is data for conversion into sign language or Braille. Specifically, the speech recognition algorithm analyzes the speech data and generates the corresponding sign language or Braille data.

[0516] Step 4:

[0517] The server uses an emotion engine to recognize user emotions from voice data. The input is preprocessed voice data, and emotions are identified using OpenCV and the Emotion API. The output is the recognized emotion data. Specifically, it analyzes the tone and intonation of the voice and identifies emotion categories (e.g., surprise, joy, sadness).

[0518] Step 5:

[0519] The server generates information based on the recognized emotion. The input is emotion data and sign language or Braille data, which are combined to generate information to provide to the user. The output is sign language or Braille information corresponding to the emotion. Specific operations include adjusting the content of the sign language or Braille based on the emotion data and setting points to emphasize.

[0520] Step 6:

[0521] The device then provides the generated sign language or Braille information to the user. The input is sign language or Braille information corresponding to the emotion, and the output is information displayed on a display or Braille display. Specifically, in the case of sign language, a sign language animation is displayed on the display, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[0522] (Application example 2)

[0523] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0524] When visually or hearing impaired people enjoy watching sports, they have difficulty understanding live broadcast audio in real time. Furthermore, because information is not provided according to the user's emotions, there is a problem in that information cannot be emphasized in response to changes in emotions. This has led to the problem that visually or hearing impaired people cannot fully enjoy watching sports.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0526] In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing a user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and then enhancing the information and converting it into sign language or Braille. This allows visually and hearing impaired people to enjoy watching sports in real time. Furthermore, because information is emphasized according to the user's emotions, a more realistic viewing experience can be provided.

[0527] "Live broadcast audio data" is audio information obtained from live broadcasts of sporting events and other live events.

[0528] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[0529] "Sign language" is a visual, gestural language used by blind people to understand spoken information.

[0530] "Braille" is a pattern of raised dots used by the visually impaired to interpret written information by touch.

[0531] An "emotion engine" is a technology that recognizes a user's emotions in real time and generates information corresponding to those emotions.

[0532] "User emotion" refers to the emotional state, such as surprise, joy, or sadness, that a user experiences while watching a sporting event.

[0533] "Emphasis on information" means displaying specific information in a way that makes it stand out according to the user's emotions.

[0534] A system for carrying out this invention is configured as follows: The server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information in accordance with the emotions recognized by the emotion engine, and means for emphasizing the information and converting it into sign language or Braille.

[0535] Program processing explanation

[0536] The server captures live audio data from the smartphone's microphone. The captured audio data is converted to text using the speech_recognition library. Next, a generative AI model is used to convert the text data into sign language or Braille. Tensorflow is used to execute the sign language to Braille conversion model.

[0537] Furthermore, the emotion engine uses the emotion_recognition library to recognize the user's emotions in real time. Based on the recognized emotions, the generative AI emphasizes specific information and converts it into sign language or Braille, enabling it to provide information tailored to the user's emotions.

[0538] Specific examples

[0539] For example, if a user is watching a live sports broadcast, the audio commentary is picked up through the smartphone's microphone. The audio data is converted into text in real time, and the text is then converted into sign language or Braille. At the same time, an emotion engine recognizes the user's emotions and highlights specific scenes or information based on emotions such as surprise or joy.

[0540] Prompt Sentence Examples

[0541] "This application converts live audio commentary of sporting events into sign language and Braille in real time, providing information to help the visually and hearing impaired enjoy watching sports. It also uses an emotion engine to recognize the user's emotions and highlights information that corresponds to those emotions."

[0542] In this way, visually and hearing impaired people can enjoy watching sports in real time, and information is emphasized according to the user's emotions, providing a more realistic viewing experience.

[0543] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0544] Step 1:

[0545] The server acquires live broadcast audio data using the smartphone's microphone. The input is audio data from the smartphone's microphone, and the output is an audio data file. This audio data is used in subsequent processing steps.

[0546] Step 2:

[0547] The server converts the acquired voice data into text using the speech_recognition library. The input is the voice data acquired in step 1, and the output is text data. This conversion allows the voice information to be treated as text information.

[0548] Step 3:

[0549] The server converts the text data into sign language and Braille using a generative AI model. The input is the text data obtained in step 2, and the output is sign language data and Braille data. This conversion uses a sign language conversion model and a Braille conversion model using Tensorflow.

[0550] Step 4:

[0551] The server recognizes the user's emotions in real time using the emotion_recognition library. The input is the text data obtained in step 2, and the output is the user's emotional data. This emotional data is used for subsequent information enhancement processing.

[0552] Step 5:

[0553] The server emphasizes specific information and converts it into sign language or Braille based on the recognized emotion data. The input is the sign language data and Braille data obtained in step 3 and the emotion data obtained in step 4, and the output is the emphasized sign language data and Braille data. This processing makes it possible to provide information according to the user's emotions.

[0554] Step 6:

[0555] The server outputs the enhanced sign language data and Braille data to the user. The input is the enhanced sign language data and Braille data obtained in step 5, and the output is the sign language and Braille information displayed on the user's device. This allows visually and hearing impaired people to enjoy watching sports in real time.

[0556] Example 3

[0557] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0558] When users with visual or hearing impairments enjoy watching sports, it is difficult to provide them with real-time information that reflects their emotions. In particular, the lack of information that takes the user's emotions into consideration limits the user's experience.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0560] In this invention, the server includes means for acquiring user emotion data, means for analyzing the emotion data, means for generating sign language or Braille data based on the analyzed emotion data, and means for outputting the sign language or Braille data, thereby enabling real-time provision of information according to the user's emotions.

[0561] "User emotion data" is data that indicates the user's emotional state, obtained from the user's facial expressions, voice, etc.

[0562] "Means for acquiring" refers to devices and software for collecting user emotion data.

[0563] "Means for analyzing" refers to a device or software for analyzing the acquired emotional data and identifying the user's emotional state.

[0564] "Means for generating" refers to devices or software for generating sign language or Braille data based on analyzed emotional data.

[0565] "Output means" refers to devices or software that provide the generated sign language or Braille data to the user.

[0566] "Sign language data" is data that includes information expressed as sign language.

[0567] "Braille data" is data that includes information expressed as Braille.

[0568] "Real-time" refers to data acquisition, analysis, generation, and output occurring without delay.

[0569] "Blind and deaf" refers to people who are visually or hearing impaired.

[0570] MODE FOR CARRYING OUT THE INVENTION

[0571] This invention is a system that provides information in real time that corresponds to the emotions of visually or hearing impaired users so that they can enjoy watching sports. Specific embodiments of this system will be described below.

[0572] System configuration

[0573] 1. Acquiring user emotion data

[0574] The user inputs their facial expressions and voice using the device's camera and microphone. The device then transmits this data to the server in real time. The hardware used includes a camera and microphone.

[0575] 2. Emotion Data Analysis

[0576] The server passes the received facial expression and voice data to the emotion engine. The emotion engine analyzes the user's facial expression and tone of voice to identify the emotion the user is feeling. For example, if the user has a sad expression, the emotion engine will recognize the emotion as "sadness." The software used includes the emotion engine.

[0577] 3. Sign Language and Braille Data Generation

[0578] The server uses a generative AI model based on the emotion data obtained from the emotion engine to generate sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if a user expresses sadness, the generative AI model generates sign language and Braille data that emphasizes sad scenes.

[0579] 4. Data Output

[0580] The server sends the generated sign language data to the device's display and sends the Braille data to the Braille display. The device displays the sign language on the display and physically generates Braille on the Braille display. The user can receive information by looking at the display or touching the Braille display. The hardware used includes the display and the Braille display.

[0581] Specific examples

[0582] As a concrete example, consider the case where a user is watching a sports game. If the user expresses sadness, the emotion engine recognizes that emotion, and the server uses a generative AI model based on that information to generate sign language or Braille data that emphasizes the sad scene. The generated data is then provided to the user via a display or Braille display.

[0583] Prompt Sentence Examples

[0584] "If a user expresses sadness while watching a sports game, recognize that emotion and generate sign language or Braille data that highlights the sadness."

[0585] In this way, the server can provide information according to the user's emotions in real time, and can support the visually or hearing impaired to enjoy watching sports. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0586] Step 1:

[0587] The user inputs their facial expressions and voice through the device's camera and microphone. The device then transmits this data to the server in real time. The input is the user's facial expression data and voice data, and the output is emotion data sent to the server.

[0588] Step 2:

[0589] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotion the user is feeling. The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the emotion engine detects features such as "drooping eyebrows" and "drooping corners of the mouth" and determines that the user is expressing sadness.

[0590] Step 3:

[0591] The server uses a generative AI model based on the emotion data obtained from the emotion engine to generate sign language and Braille data. The input is analyzed emotion information, and the output is generated sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille data according to the emotion. For example, if a user expresses sadness, the generative AI model generates sign language and Braille data that emphasizes sad scenes.

[0592] Step 4:

[0593] The server sends the generated sign language data to the terminal's display and sends the Braille data to the Braille display. The input is the generated sign language data and Braille data, and the output is the sign language displayed on the display and Braille formed on the Braille display. The terminal displays the sign language on the display and physically forms Braille on the Braille display. The user can understand the sign language by looking at the display and read Braille by touching the Braille display.

[0594] In this way, the server can provide information in real time that corresponds to the user's emotions, helping visually or hearing impaired people to enjoy watching sports.

[0595] (Application example 3)

[0596] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0597] When users with visual or hearing impairments enjoy watching sports, they face the challenge of being unable to provide real-time information.In addition, the inability to provide information that responds to the user's emotions limits the user experience.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, means for generating sign language or Braille data according to the emotions, means for displaying the sign language data on a display, and means for outputting the Braille data to a Braille display. This enables users with visual or hearing impairments to enjoy watching sports in real time, and also makes it possible to provide information according to the user's emotions.

[0599] "Live broadcast audio data" refers to audio data that conveys the progress of a sporting event or other live event in real time.

[0600] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[0601] "Sign language" is a visual, gestural language used by deaf people to communicate.

[0602] "Braille" is a writing system using raised dots that is used by the visually impaired to read by touch.

[0603] A "display" is a device for visually displaying information.

[0604] A "Braille display" is a device that physically creates Braille characters that can be read by the visually impaired through touch.

[0605] "Means for recognizing user emotions" refers to technology that analyzes and recognizes emotions from the user's facial expressions, voice, etc.

[0606] "Means for generating sign language data" refers to technology that converts input information into sign language format.

[0607] "Means for generating Braille data" refers to technology that converts input information into Braille format.

[0608] "Means for displaying sign language data on a display" refers to technology for displaying the generated sign language data on a display.

[0609] "Means for outputting Braille data to a Braille display" refers to technology for outputting the generated Braille data to a Braille display.

[0610] A system for implementing the present invention has the following configuration.

[0611] System Program

[0612] The system includes a means for capturing live broadcast audio data, a means for analyzing the audio data using generation AI and converting it into sign language or Braille in real time, a means for outputting sign language or Braille, a means for recognizing the user's emotions, a means for generating sign language or Braille data according to the emotions, a means for displaying the sign language data on a display, and a means for outputting Braille data to a Braille display.

[0613] Processing Description

[0614] Hardware used

[0615] Smartphone camera

[0616] Smartphone displays

[0617] Braille display

[0618] Software used

[0619] OpenCV: Camera image capture and display

[0620] EmotionRecognizer: Emotion recognition model

[0621] SignLanguageGenerator: A sign language generation model

[0622] BrailleGenerator: A Braille Generation Model

[0623] Data processing and calculation

[0624] The server captures the user's face with the smartphone's camera and inputs the image into EmotionRecognizer to recognize the user's emotions. Based on the recognized emotions, SignLanguageGenerator generates sign language data and BrailleGenerator generates Braille data. The generated sign language data is displayed on the smartphone's display, and the Braille data is output to a Braille display.

[0625] Specific examples

[0626] For example, if a user expresses sadness while watching a sports game, the system will recognize that emotion, highlight the sadness, and convert it into sign language or Braille. A scene in which a player on a losing team is crying will be explained in detail in sign language or Braille.

[0627] Prompt Sentence Examples

[0628] Create a program that recognizes the user's emotions and generates sign language or Braille data according to the emotion. If the user expresses sadness, develop an application that provides information according to that emotion in sign language or Braille.

[0629] The above is an embodiment of the present invention. This system enables users with visual or hearing impairments to enjoy watching sports in real time, and also provides information that responds to the user's emotions.

[0630] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0631] Step 1:

[0632] The server captures the user's face using the smartphone camera. The input is the camera image, and the output is the captured image data. This image data is used as preprocessing for emotion recognition.

[0633] Step 2:

[0634] The server inputs the captured image data into EmotionRecognizer to recognize the user's emotions. The input is image data, and the output is recognized emotion data. EmotionRecognizer extracts facial features from the image data and classifies the emotion.

[0635] Step 3:

[0636] The server generates sign language data using SignLanguageGenerator based on the recognized emotion data. The input is emotion data and the output is sign language data. SignLanguageGenerator generates sign language actions corresponding to the emotion data.

[0637] Step 4:

[0638] The server generates Braille data using the BrailleGenerator based on the recognized emotion data. The input is emotion data and the output is Braille data. The BrailleGenerator generates Braille patterns corresponding to the emotion data.

[0639] Step 5:

[0640] The server displays the generated sign language data on the smartphone's display. The input is sign language data, and the output is sign language displayed on the display. The display provides the user with a visual representation of the sign language actions.

[0641] Step 6:

[0642] The server outputs the generated braille data to a braille display. The input is braille data, and the output is a braille display on the braille display. The braille display physically creates braille patterns that can be read by the user through touch.

[0643] Step 7:

[0644] When a user expresses emotion while watching a sporting event, the server recognizes that emotion in real time and generates and outputs appropriate sign language or Braille data, allowing the user to receive information according to their emotion in real time.

[0645] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0646] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0647] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0648] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0649] [Second embodiment]

[0650] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0651] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0652] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32.

[0653] The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0654] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0655] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0656] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0657] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0658] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0659] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0660] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0661] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0662] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0663] "Example 1"

[0664] In one embodiment of the present invention, a microphone and audio input device are provided to capture live audio data from a sports broadcast. This audio input device can capture live audio data in real time. The captured audio data is then sent to the AI ​​generation system.

[0665] "Example 2"

[0666] The generative AI analyzes the captured voice data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the voice data. This generative AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[0667] "Example 3"

[0668] The generated sign language or Braille data is provided to the user through an output device. Examples of output devices include a display for displaying sign language and a Braille display for outputting Braille. A specific example is a Braille display, which physically generates Braille based on the generated Braille data, allowing visually impaired people to read it by touch.

[0669] The processing flow of each embodiment will be described below.

[0670] "Example 1"

[0671] Step 1: Provide a microphone or audio input device that captures live broadcast audio data of a sports event. This audio input device can capture live broadcast audio data in real time.

[0672] Step 2: The captured voice data is sent to the generation AI, which is trained using AI techniques such as deep learning to achieve highly accurate conversion.

[0673] "Example 2"

[0674] Step 1: The generation AI analyzes the captured voice data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the voice data.

[0675] Step 2: The generated sign language or Braille data is provided to the user through an output device, such as a display for displaying sign language or a Braille display for outputting Braille.

[0676] "Example 3"

[0677] Step 1: As a concrete example, a Braille display physically forms Braille characters based on the generated Braille data, allowing visually impaired people to read them tactilely.

[0678] Example 1

[0679] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] People with visual or hearing impairments have difficulty enjoying watching sports. In particular, they have difficulty understanding audio information from live broadcasts in real time, which significantly reduces the enjoyment of watching sports.

[0681] The specification processing by the specification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for importing live broadcast audio data, a means for transmitting the live broadcast audio data to the generative AI model, and a means for the generative AI model to analyze the audio data and convert it into text. This enables people with visual or hearing impairments to enjoy watching sports in real time.

[0682] "Live broadcast audio data" is audio information provided in real time by a commentator at an event such as a sports game.

[0683] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates a specific output.

[0684] A "means" is a method or device used to achieve a particular purpose.

[0685] "Convert to text" is the process of analyzing audio data and converting it into text information.

[0686] "Output" refers to providing the processed data in a display, audio, or other format for presentation to a user.

[0687] MODE FOR CARRYING OUT THE INVENTION

[0688] This invention is a system that captures live audio data from sports broadcasts in real time and sends that data to a generative AI model. Specifically, the system acquires audio data using a microphone or audio input device, and sends the data to a generative AI model, where it performs various data processing and calculations.

[0689] Hardware and software used

[0690] Hardware: Microphone, audio input device

[0691] Software: Generative AI models (e.g., OpenAI's GPT-4)

[0692] System Overview

[0693] 1. Audio data capture:

[0694] A user broadcasts a live sports event.

[0695] A microphone or audio input device connected to the terminal captures audio data from the live broadcast in real time.

[0696] 2. Sending audio data:

[0697] The captured voice data is transmitted from the terminal to the server.

[0698] The server sends the received voice data to the generative AI model.

[0699] 3. Data analysis and processing:

[0700] The generative AI model analyzes the received voice data and performs the necessary data processing and calculations.

[0701] For example, it converts audio data into text, extracts important events, and generates real-time highlights.

[0702] 4. Result output:

[0703] The server receives the results from the generative AI model and provides feedback to the user.

[0704] Specifically, the generated text is sent to the user's terminal, and the terminal displays the text.

[0705] Specific examples

[0706] Example 1:

[0707] A user is broadcasting a live soccer match.

[0708] The microphone picks up the sound of "Goal!"

[0709] The terminal transmits this audio data to the server.

[0710] The server sends the voice data to the generative AI model, which then generates the text "A goal has been scored!"

[0711] Example 2:

[0712] A user is broadcasting a basketball game.

[0713] The microphone picks up the audio of "Three-point shot successful!"

[0714] The terminal transmits this audio data to the server.

[0715] The server sends the voice data to the generative AI model, which then generates the text "The three-point shot was successful!"

[0716] Prompt Sentence Examples

[0717] Prompt 1:

[0718] "Please analyze this audio data and extract the key events. Audio data: 'Goal!'"

[0719] Prompt Statement 2:

[0720] "Please convert this audio data to text. Audio data: 'Three-pointer made!'"

[0721] The above is an embodiment of the present invention.

[0722] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0723] Step 1:

[0724] A user is watching a sports game and broadcasting live commentary. A microphone and audio input device connected to the device captures the audio data of the live broadcast in real time. Specifically, when the user shouts "Goal!", the microphone captures the audio and converts it into digital data. The input is the user's voice, and the output is digital audio data.

[0725] Step 2:

[0726] The terminal sends the captured voice data to the server. Specifically, the terminal compresses the voice data and sends it to the server using the TCP / IP protocol. The input is digital voice data, and the output is the voice data sent to the server.

[0727] Step 3:

[0728] The server sends the received voice data to the generative AI model. Specifically, the server sends the voice data to the API of the generative AI model and waits for the analysis results. The input is the voice data sent to the server, and the output is the voice data sent to the generative AI model.

[0729] Step 4:

[0730] The generative AI model analyzes the voice data and performs the necessary data processing and calculations. Specifically, the generative AI model converts the voice data into text and generates the text "A goal has been scored!" The input is the voice data sent to the generative AI model, and the output is the generated text.

[0731] Step 5:

[0732] The server receives the results from the generative AI model and provides feedback to the user. Specifically, the server sends the generated text to the user's device, which then displays the text. The input is the generated text, and the output is the text displayed on the user's device.

[0733] (Application example 1)

[0734] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0735] In the past, when watching sports, it was difficult for people with visual or hearing impairments to understand the progress of the game and important highlights in real time. Furthermore, even for general spectators, there were limited means to obtain important highlights and statistical information of the game in real time. This limited the enjoyment of watching the game.

[0736] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0737] In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the converted commentary, and means for outputting the generated highlights and statistical information. This enables visually or hearing impaired people to obtain information in real time to enjoy watching sports, and also enables general spectators to obtain important highlights and statistical information of the game in real time.

[0738] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports or events.

[0739] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate specific outputs.

[0740] "Sign language" is a language that uses hand and finger movements to communicate with people who are deaf or hard of hearing.

[0741] Braille is a tactile representation of letters made up of raised dots that is used by the visually impaired to read.

[0742] "Means for converting to text" refers to technology that analyzes voice data and converts it into corresponding text information.

[0743] "Highlights" refers to important moments or events in sports or events.

[0744] "Statistical Information" refers to numerical data and analytical results related to a match or event.

[0745] "Output means" refers to technology such as a display device or audio device for providing the generated information to the user.

[0746] A system for implementing this invention includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the textual commentary content, and means for outputting the generated highlights and statistical information.

[0747] Hardware and software used

[0748] Hardware: Smartphone (with built-in microphone)

[0749] Software: Python, SpeechRecognition library (voice recognition), OpenAI API (generative AI)

[0750] Processing flow

[0751] 1. Audio data capture:

[0752] The smartphone microphone is used to capture live audio data in real time. The user launches the smartphone application and starts voice input.

[0753] 2. Audio to text conversion:

[0754] The captured audio data is converted to text using the SpeechRecognition library. The server sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data.

[0755] 3. Generative AI analysis:

[0756] The transcribed commentary is sent to a generator AI using the OpenAI API, which analyzes the input text data and generates key highlights and statistics.

[0757] 4. Output of information:

[0758] The generated highlights and statistics are displayed in real time on the user's smartphone, allowing the user to keep track of key moments and statistics of the match in real time.

[0759] Specific examples

[0760] When a user is watching a soccer match, they launch a smartphone app and capture the audio from the live broadcast. The app converts the audio into text and displays important events such as "Goal!" and "Foul!" in real time. It also automatically generates and displays highlights and statistics from the match (e.g., number of shots, ball possession).

[0761] Prompt Sentence Examples

[0762] Generate highlights of the following sports plays:

[0763] "30 minutes into the first half, Team A scored a goal. Team B's defense collapsed. Team A continued to attack in the second half."

[0764] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0765] Step 1:

[0766] A user launches a smartphone application to capture the audio of a live broadcast. The user uses the smartphone's microphone to collect audio data in real time. The input is the audio data of the live broadcast, and the output is a stream of audio data.

[0767] Step 2:

[0768] The device converts the captured audio data into text using the SpeechRecognition library. The device sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data. The input is a stream of audio data, and the output is text data.

[0769] Step 3:

[0770] The server sends the textual commentary to the generation AI using the OpenAI API. The server analyzes the input text data and generates important highlights and statistics. The input is text data, and the output is the generated highlights and statistics.

[0771] Step 4:

[0772] The server sends the generated highlights and statistical information to the user's smartphone in real time, where the user can view the information on the smartphone screen. The input is the generated highlights and statistical information, and the output is the information displayed on the smartphone screen.

[0773] Step 5:

[0774] Users can understand key moments and statistical information of a match in real time through a smartphone application. Specifically, users check the information on the screen and understand the progress of the match. The input is the information displayed on the smartphone screen, and the output is the user's understanding.

[0775] Example 2

[0776] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0777] People with visual or hearing impairments have difficulty understanding audio information in real time. In particular, the accuracy and speed of converting audio information into sign language or Braille is insufficient, which can delay the transmission of information or lead to misunderstandings. This makes it difficult for people with visual or hearing impairments to receive information accurately and quickly.

[0778] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0779] In this invention, the server includes a means for capturing voice data, a means for converting the voice data into text data using voice recognition software, and a means for inputting the text data into a generative AI model to generate sign language data or Braille data, thereby enabling visually or hearing impaired people to receive voice information in sign language or Braille in real time.

[0780] "Audio data" refers to data in which audio is recorded in digital format.

[0781] "Speech recognition software" is software that analyzes voice data and converts it into text data.

[0782] "Text data" refers to data in which character information is recorded in digital format.

[0783] A "generative AI model" is an artificial intelligence model that is trained using deep learning technology and generates sign language data or Braille data based on text data.

[0784] "Sign language data" refers to data that records sign language movements in digital format.

[0785] "Braille data" refers to data that records Braille information in digital format.

[0786] "Sign language animation software" is software that visually displays sign language movements based on sign language data.

[0787] A "Braille display" is a device that physically displays Braille based on Braille data.

[0788] This invention is a system that enables people with visual or hearing impairments to receive audio information in real time as sign language or Braille. A specific embodiment of this system will be described below.

[0789] System configuration

[0790] Server Configuration

[0791] The server includes a means for capturing voice data, a means for converting the voice data into text data using voice recognition software, and a means for converting the text data into sign language data or Braille data using a generative AI model. The voice recognition software uses a general voice recognition API (e.g., Google Cloud Speech-to-Text API). The generative AI model is a model trained using deep learning technology (e.g., a model using TensorFlow or PyTorch).

[0792] Device configuration

[0793] The terminal includes a means for visually displaying the sign language data sent from the server using sign language animation software (e.g., SignAll), and a means for outputting Braille data to a Braille display (e.g., Focus Blue). The terminal also includes a microphone for the user to input voice data.

[0794] User operations

[0795] The user inputs voice data by speaking into the device's microphone. For example, they might say "hello." The device then sends this voice data to the server. The server converts the voice data into text data and generates sign language or Braille data using a generative AI model. The generated data is then sent to the device and displayed on the display as sign language animation or output to a Braille display.

[0796] Specific examples

[0797] Example 1: Conversion to sign language

[0798] When a user says "Good morning," the device sends the voice data to the server. The server converts the voice data into text and inputs the text information "Good morning" into the generative AI model. The generative AI model generates sign language data corresponding to "Good morning" and creates a sign language animation using sign language animation software. The device displays this sign language animation on the display, allowing the user to visually confirm the sign language for "Good morning."

[0799] Example 2: Conversion to Braille

[0800] When a user says "Thank you," the device sends the voice data to the server. The server converts the voice data into text and inputs the text information "Thank you" into the generative AI model. The generative AI model generates Braille data corresponding to "Thank you" and sends it to the Braille display. The Braille display connected to the device displays "Thank you" in Braille, allowing the user to read the Braille with their sense of touch.

[0801] Prompt Sentence Examples

[0802] "If a user says 'Good morning,' how would your generative AI model convert that into sign language or Braille?"

[0803] By inputting this prompt into a generative AI model, a detailed explanation of the specific conversion process can be obtained.

[0804] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0805] Step 1:

[0806] The user inputs voice data.

[0807] The user speaks into the device's microphone. For example, they say "hello." The device captures this speech as digital audio data. The input is the user's voice, and the output is digital audio data.

[0808] Step 2:

[0809] The terminal transmits the voice data to the server.

[0810] The device transmits the captured audio data over the Internet to a server, where it is encoded in an appropriate format (e.g., WAV or MP3). The input is the digital audio data, and the output is the audio data transmitted to the server.

[0811] Step 3:

[0812] The server converts the voice data into text data.

[0813] The server converts the received voice data into text data using voice recognition software (for example, a general voice recognition API). For example, voice data saying "Hello" is converted into text data saying "Hello." The input is voice data, and the output is text data.

[0814] Step 4:

[0815] The server inputs the text data into the generative AI model.

[0816] The server inputs the converted text data into a generative AI model, which is trained using deep learning techniques. The input is text data, and the output is sign language data or Braille data.

[0817] Step 5:

[0818] A generative AI model generates sign language and Braille data.

[0819] A generative AI model generates sign language data or Braille data based on input text data. For example, the text data "Hello" is converted into sign language animation data or Braille data. The input is text data, and the output is sign language data or Braille data.

[0820] Step 6:

[0821] The server transmits the generated data to the terminal.

[0822] The server sends the generated sign language or braille data to the device. The data is encoded in an appropriate format (e.g., JSON or XML). The input is the sign language or braille data, and the output is the data sent to the device.

[0823] Step 7:

[0824] The device displays sign language animations and outputs braille data to a braille display.

[0825] The terminal displays the received sign language data on a display using sign language animation software (e.g., sign language animation software). In addition, the Braille data is output to a Braille display (e.g., a Braille display) connected to the terminal. The user can visually check the sign language animation displayed on the display and tactilely read the Braille displayed on the Braille display. The input is sign language data or Braille data, and the output is sign language animation or Braille.

[0826] (Application example 2)

[0827] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0828] People with visual or hearing impairments have difficulty understanding voice instructions and warnings in real time during security work, which can prevent them from working efficiently and lead to a decline in the quality of security.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting audio data into text, means for converting the text into sign language and Braille, and means for providing the sign language and Braille to visually impaired and hearing impaired persons. This enables visually impaired and hearing impaired persons to understand audio instructions and warnings in real time and perform security operations efficiently.

[0830] "Live broadcast audio data" refers to audio information about events or situations occurring in real time.

[0831] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[0832] "Sign language" refers to a means of communicating linguistic information using visual gestures and movements.

[0833] "Braille" refers to a writing system that uses raised dots to enable visually impaired people to read written information using their sense of touch.

[0834] "Audio data" refers to information that records sound in digital form.

[0835] "Text" refers to textual information.

[0836] "Blind people" refers to people who are visually impaired.

[0837] "Deaf" refers to people who have hearing impairments.

[0838] "Security work" refers to work to protect the safety of facilities and people.

[0839] "Real-time" refers to processing and transmission occurring immediately without delay.

[0840] A system for carrying out the present invention converts voice data into sign language or Braille in real time and provides the converted data to visually impaired or hearing impaired people. A specific embodiment of the system will be described below.

[0841] System configuration

[0842] Hardware

[0843] Smartphone: Equipped with a microphone for capturing voice data and a display for showing the conversion results.

[0844] Server: Provides the computing resources for analyzing and converting audio data.

[0845] software

[0846] Speech Recognition Library: Use the speech_recognition library to convert audio data to text.

[0847] Sign Language Conversion Library: Use the hand_gesture_recognition library to convert text to sign language.

[0848] Braille conversion library: Use the braille_converter library to convert text to braille.

[0849] Generative AI models: Use deep learning models to analyze audio data and convert it into sign language or Braille.

[0850] System Operation

[0851] Acquiring audio data

[0852] The user inputs voice data using the smartphone's microphone. For example, a security guard may input "There is a suspicious person here."

[0853] Analysis of audio data

[0854] The server converts the captured audio data into text using the speech_recognition library, which serves as the basis for subsequent sign language and braille conversion.

[0855] Conversion to sign language and braille

[0856] The server uses generative AI models to convert text to sign language and braille using the hand_gesture_recognition library for sign language conversion and the braille_converter library for braille conversion.

[0857] Providing results

[0858] The converted sign language and Braille are then displayed on the smartphone's display, allowing visually and hearing impaired people to understand voice instructions and warnings in real time.

[0859] Specific examples

[0860] For example, if a security guard says "There's a suspicious person here," the system converts this speech into text and then into sign language and Braille, which are then displayed on the smartphone screen, making the information accessible to security guards who are visually or hearing impaired.

[0861] Prompt Sentence Examples

[0862] Enter the voice data. Example: "There is a suspicious person."

[0863] In this way, visually and hearing impaired people can understand voice instructions and warnings in real time and perform security tasks efficiently.

[0864] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0865] Step 1:

[0866] The user inputs voice data using the smartphone's microphone, which is temporarily stored in the smartphone's internal memory.

[0867] Step 2:

[0868] The smartphone sends the saved voice data to the server. The server receives the voice data and converts it into text using the speech_recognition library. The input is voice data and the output is text data.

[0869] Step 3:

[0870] The server analyzes the text data using a generative AI model. This analysis converts the text data into sign language and Braille. The input is text data, and the output is sign language and Braille data.

[0871] Step 4:

[0872] The server converts the sign language data into sign language using the hand_gesture_recognition library. The input is text data and the output is sign language data.

[0873] Step 5:

[0874] The server converts the braille data into braille using the braille_converter library. The input is text data and the output is braille data.

[0875] Step 6:

[0876] The server then sends the converted sign language and Braille data to the smartphone, which then displays the received data on its display, enabling visually and hearing impaired people to understand voice instructions and warnings in real time.

[0877] Example 3

[0878] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0879] The purpose of this system is to solve the problem that users with visual or hearing impairments have difficulty converting text data into sign language or Braille and using information in real time. In particular, there is a need for a system that can quickly and accurately convert text data entered by users into sign language or Braille and provide it through an appropriate output device.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0881] In this invention, the server includes a means for a user to input text data, a means for the server to receive the input data and send a prompt to the generative AI model, a means for the generative AI model to generate sign language or Braille data, a means for the server to send the generated data to an output device, and a means for the terminal to provide the sign language or Braille to the user through the output device, thereby enabling visually or hearing impaired users to quickly and accurately convert input text data into sign language or Braille and use information in real time.

[0882] "User" refers to a person who uses the system to input text data and receive the information converted into sign language or Braille.

[0883] "Server" refers to a computer system whose role is to receive input data from a user, send prompts to a generative AI model, and send the generated sign language or Braille data to an output device.

[0884] A "generative AI model" refers to an artificial intelligence model that generates sign language or Braille data based on text data from the user.

[0885] A "prompt sentence" refers to a sentence that instructs a generative AI model to convert text data into sign language or Braille.

[0886] An "output device" is a device for providing the generated sign language or Braille data to a user, and includes a sign language display, a Braille display, and the like.

[0887] "Sign language display" refers to a display device for visually displaying generated sign language data.

[0888] "Braille display" refers to a device that physically renders generated Braille data so that it can be read by the visually impaired through touch.

[0889] "Text data" refers to textual information that a user inputs into a system.

[0890] "Real-time" means that text data entered by the user is instantly converted into sign language or Braille and provided almost simultaneously through an output device.

[0891] MODE FOR CARRYING OUT THE INVENTION

[0892] This invention is a system that enables visually or hearing impaired users to convert text data into sign language or Braille and access information in real time. A specific embodiment of this system will be described below.

[0893] System configuration

[0894] The system consists of the following major components:

[0895] 1. User terminal: A device that allows a user to input text data. Examples include PCs, smartphones, and tablets.

[0896] 2. Server: Receives input data from the user, sends prompts to the generative AI model, and sends the generated sign language or Braille data to the output device. Use a high-performance server (e.g., a server equipped with an NVIDIA GPU).

[0897] 3. Generative AI model: An artificial intelligence model for generating sign language or Braille data based on text data from users. It uses deep learning frameworks such as TensorFlow and PyTorch.

[0898] 4. Output device: A device that presents the generated sign language or Braille data to the user, including a sign language display or a Braille display.

[0899] System Operation

[0900] 1. The user enters text data.

[0901] A user inputs text data through the system's interface, for example, by typing "hello" using a web browser or a dedicated application.

[0902] 2. The server receives the input data and sends a prompt to the generative AI model.

[0903] The server receives the text data sent by the user. Based on the received data, it creates a prompt sentence and sends it to the generative AI model. For example, it sends the prompt sentence "Please convert the following text into sign language: Hello" to the generative AI model.

[0904] 3. The generative AI model generates sign language and Braille data.

[0905] The generative AI model analyzes the prompt received from the server and generates corresponding sign language or Braille data, such as sign language animation data or Braille sequence data.

[0906] 4. The server sends the generated data to an output device.

[0907] The server sends the sign language and Braille data received from the generative AI model to an output device, for example, sending sign language animation data to a sign language display and Braille data to a Braille display.

[0908] 5. The terminal provides sign language or Braille to the user through an output device.

[0909] The terminal receives the data sent from the server and provides it to the user through the output device. The sign language display shows the sign language animation, and the braille display physically generates braille. The user can see the sign language display visually and read the braille display tactilely.

[0910] Specific examples

[0911] For example, when the user inputs the text "Thank you," the specific operation is as follows.

[0912] 1. User: Enters "Thank you" into the input field in a web browser and clicks the send button.

[0913] 2. Server: Receives the text data "Thank you" from the user and sends the prompt sentence "Please convert the following text into sign language: Thank you" to the generative AI model.

[0914] 3. Generative AI model: Analyzes the prompt and generates sign language animation data corresponding to "thank you."

[0915] 4. Server: Sends the generated sign language animation data to the sign language display.

[0916] 5. Terminal: The sign language display shows the sign language animation, which the user can visually confirm.

[0917] In this way, the user can easily convert text into sign language or Braille and use it through an output device. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0918] Step 1:

[0919] The user inputs text data.

[0920] A user inputs text data using a web browser or a dedicated application. For example, the user inputs "Hello" and clicks the send button. The input data is sent from the user's device to the server.

[0921] Step 2:

[0922] The server receives the input data and sends prompts to the generative AI model.

[0923] The server receives text data sent by the user. Based on the received data, it creates a prompt sentence and sends it to the generative AI model. For example, the prompt sentence "Please convert the following text into sign language: Hello" is sent to the generative AI model. The input is the user's text data, and the output is the prompt sentence to the generative AI model.

[0924] Step 3:

[0925] A generative AI model generates sign language and Braille data.

[0926] The generative AI model analyzes the prompt received from the server and generates the corresponding sign language or Braille data. For example, it generates sign language animation data or Braille sequence data. The input is the prompt, and the output is the sign language or Braille data.

[0927] Step 4:

[0928] The server sends the generated data to an output device.

[0929] The server sends the sign language or Braille data received from the generative AI model to an output device. For example, it sends sign language animation data to a sign language display and Braille data to a Braille display. The input is the generated sign language or Braille data, and the output is the data sent to the output device.

[0930] Step 5:

[0931] The terminal provides sign language or Braille to the user through an output device.

[0932] The terminal receives data sent from the server and provides it to the user through the output device. The sign language display shows sign language animations, and the braille display physically produces braille. The user can see the sign language display visually and read the braille display tactilely. The input is sign language or braille data from the server, and the output is the provision of sign language or braille to the user.

[0933] (Application example 3)

[0934] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0935] People with visual or hearing impairments have difficulty receiving real-time information when watching sports or obtaining security information. While there is a need for rapid communication of information in sign language or Braille in emergencies, current systems are inadequate.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing security camera footage, analyzing it using a sign language recognition model, and recognizing the sign language, means for displaying the recognized sign language on the footage, and means for outputting security information on a Braille display. This enables people with visual or hearing impairments to watch sports and receive security information in real time.

[0937] "Live broadcast audio data" refers to data that provides audio information about the progress of sporting events and other live events.

[0938] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[0939] "Sign language" is a visual, gestural language used by deaf people to communicate.

[0940] "Braille" is a writing system using raised dots that is used by the visually impaired to read by touch.

[0941] A "sign language recognition model" is a machine learning model that analyzes sign language movements from camera footage and converts them into the corresponding language.

[0942] A "security camera" is a camera device for monitoring a specific area and acquiring video data.

[0943] A "Braille display" is a device that displays Braille characters using electronically controlled raised dots.

[0944] "Security information" is information relating to abnormalities or emergencies within the monitored area.

[0945] A system for implementing this invention is configured as follows: The server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing security camera footage, analyzing it using a sign language recognition model, and recognizing the sign language, means for displaying the recognized sign language on the footage, and means for outputting security information on a Braille display.

[0946] Hardware and software used

[0947] Hardware:

[0948] Smartphone camera

[0949] Braille displays (e.g. BrailleDisplay)

[0950] security cameras

[0951] software:

[0952] OpenCV: Acquiring and displaying camera images

[0953] HandSignRecognition: A sign language recognition model

[0954] BrailleDisplay: A library for controlling braille displays

[0955] Data processing and calculation

[0956] 1. Importing live broadcast audio data:

[0957] The server captures play-by-play audio data that provides audio updates on the progress of sporting events and other live events.

[0958] 2. Generative AI analysis:

[0959] The server analyzes the captured live broadcast audio data using a generation AI and converts it into sign language or Braille in real time.

[0960] 3. Sign language and Braille output:

[0961] The server outputs the generated sign language or Braille through a display or Braille display.

[0962] 4. Security camera capture:

[0963] The server captures the security camera footage and analyzes it using a sign language recognition model.

[0964] 5. Sign Language Recognition and Display:

[0965] The server recognizes the sign language in the video using a sign language recognition model and displays the recognized sign language on the video.

[0966] 6. Outputting security information to a braille display:

[0967] The server outputs security information to a braille display.

[0968] Specific examples

[0969] Sports viewing:

[0970] Users with visual or hearing impairments can receive information from live broadcasts of sporting events by converting audio data into sign language or Braille in real time through a display or Braille display.

[0971] Obtaining security information:

[0972] The security camera captures the video, recognizes the sign language using a sign language recognition model, and displays the recognized sign language on the video. Security information is also output to a Braille display, allowing visually impaired people to receive the information tactilely.

[0973] Prompt Sentence Examples

[0974] "Create a program that analyzes security camera footage, displays emergency information in sign language, and outputs security information on a braille display."

[0975] The above is an embodiment of the present invention.

[0976] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0977] Step 1:

[0978] The server captures play-by-play audio data that provides audio information about the progress of a sporting event or other live event. The input is the play-by-play audio data, and the output is the captured audio data, which is used for subsequent analysis.

[0979] Step 2:

[0980] The server analyzes the captured live broadcast audio data using a generation AI and converts it into sign language or Braille in real time. The input is the live broadcast audio data, and the output is sign language data and Braille data. The generation AI analyzes the audio data and generates the corresponding sign language or Braille.

[0981] Step 3:

[0982] The server outputs the generated sign language and braille data through a display or braille display. The input is sign language data and braille data, and the output is sign language displayed on the display and braille displayed on the braille display. This allows users to receive information even if they are visually or hearing impaired.

[0983] Step 4:

[0984] The server captures the security camera footage and analyzes it using a sign language recognition model. The input is the security camera footage and the output is the recognized sign language data. The sign language recognition model analyzes the footage and recognizes the sign language movements.

[0985] Step 5:

[0986] The server displays the recognized sign language on the video. The input is the recognized sign language data, and the output is the sign language displayed on the video. This allows the user to understand the security camera video in sign language.

[0987] Step 6:

[0988] The server outputs security information to a braille display. The input is security information, and the output is a braille display on the braille display. This allows the user to receive security information tactilely.

[0989] The above are the specific processing steps for carrying out the present invention.

[0990] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0991] "Example 1"

[0992] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes a user's emotions and generates information based on those emotions to help visually or hearing-impaired people enjoy watching sports more. Specifically, the system infers emotions from the user's facial expressions, tone of voice, and word choice, and analyzes live broadcast audio data based on those emotions. For example, if a user expresses joy, the system recognizes that emotion and converts it into sign language or Braille, emphasizing the more exciting scenes.

[0993] "Example 2"

[0994] The emotion engine also recognizes emotions in real time and generates information based on those emotions. This makes it possible to provide information tailored to the user's real-time emotions, helping visually and hearing impaired people enjoy watching sports. For example, if a user expresses surprise, the system recognizes that emotion, highlights the surprise, and converts it into sign language or Braille.

[0995] "Example 3"

[0996] Furthermore, the emotion engine generates information based on the user's emotions and outputs that information in real time. This makes it possible to instantly provide information tailored to the user's emotions so that visually and hearing impaired people can enjoy watching sports. Specifically, if a user expresses sadness, the system recognizes that emotion, highlights sad scenes, and converts them into sign language or Braille.

[0997] The processing flow of each embodiment will be described below.

[0998] "Example 1"

[0999] Step 1: Estimate the user's emotions from their facial expressions, tone of voice, choice of words, etc.

[1000] Step 2: Analyze the live broadcast audio data according to the estimated emotions.

[1001] Step 3: Based on the analysis results, highlight the more exciting scenes and convert them into sign language or Braille.

[1002] "Example 2"

[1003] Step 1: Recognize user emotions in real time.

[1004] Step 2: Analyze the live broadcast audio data according to the recognized emotions.

[1005] Step 3: Based on the analysis results, highlight surprising scenes and convert them into sign language or Braille.

[1006] "Example 3"

[1007] Step 1: Recognize user emotions in real time and generate information based on those emotions.

[1008] Step 2: Output the generated information in real time.

[1009] Step 3: Based on the output information, highlight the sad scenes and convert them into sign language or Braille.

[1010] Example 1

[1011] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1012] When viewing sports, visually or hearing impaired users face the challenge of having difficulty understanding live audio information in real time. Another problem is that the enjoyment of watching a sporting event is not fully conveyed because information cannot be provided in response to the user's emotions.

[1013] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for importing live broadcast audio data, a means for analyzing the live broadcast audio data using a generative AI model and recognizing the user's emotion, and a means for converting the live broadcast audio data into sign language or Braille in real time according to the user's emotion. This makes it possible to provide information in real time so that users with visual or hearing impairments can enjoy watching sports.

[1014] "Live broadcast audio data" refers to data that provides real-time audio information about the progress of sporting events and other live events.

[1015] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to analyze data and generate information for a specific purpose.

[1016] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, tone of voice, choice of words, etc. to estimate their emotional state.

[1017] "Means for converting into sign language or Braille" refers to software or hardware for converting audio data or text data into sign language video or Braille information.

[1018] "Means for outputting sign language or Braille" refers to an output device such as a display or Braille display that provides the generated sign language video or Braille information to the user.

[1019] An "exciting scene" is an important moment or play that creates particular excitement and emotion in watching sports.

[1020] This invention is a system that allows visually or hearing impaired users to enjoy watching sports. The system captures live broadcast audio data in real time, recognizes the user's emotions, and converts information corresponding to those emotions into sign language or Braille and provides it.

[1021] Hardware and software used

[1022] Hardware

[1023] Microphones: Installed in stadiums and event venues, they capture live audio data in real time.

[1024] Camera: Used to capture the user's facial expressions.

[1025] Terminal: A device that processes audio and video data and sends it to a server.

[1026] Braille display: An output device for providing braille information to a user.

[1027] software

[1028] Generative AI model: An algorithm that analyzes voice data and recognizes user emotions.

[1029] Emotion engine: Software that estimates emotions from a user's facial expressions, tone of voice, choice of words, etc.

[1030] Sign language translation software: Software for converting audio and text data into sign language video.

[1031] Braille conversion software: Software for converting audio and text data into Braille information.

[1032] Specific operation of the system

[1033] Importing audio data

[1034] The device uses high-sensitivity microphones installed in the stadium to capture live audio data in real time, for example, capturing the audio of the commentator shouting "Goal!"

[1035] Sending audio data

[1036] The device then transmits the captured audio data over the internet to a cloud-based generative AI model, where it is compressed and sent to a server using a secure communication protocol.

[1037] User emotion recognition

[1038] The server monitors the user's connection to the system via a camera and microphone. The emotion engine analyzes the user's facial expressions (e.g., smile or surprise) and tone of voice (e.g., excitement) in real time to estimate the user's emotions.

[1039] Emotion-based data analysis

[1040] If the server estimates that the user's emotion is joy, it will highlight exciting scenes from the live broadcast audio data, for example, by identifying and extracting the moments of goals and important plays.

[1041] Information generation and provision

[1042] The server converts the extracted exciting scenes into sign language and Braille. Specifically, it uses sign language translation software to generate sign language videos and Braille conversion software to display them on a Braille display. The generated information is sent to the user's device and displayed in real time.

[1043] Examples and prompts

[1044] Specific examples

[1045] If a user is watching a soccer match and shows a facial expression of joy when a goal is scored, the system will recognize that emotion and convert the commentary of the goal into sign language or Braille, highlighting it.

[1046] Prompt Sentence Examples

[1047] "Capture the audio of a live broadcast of a soccer game in real time, the moment a goal is scored, recognize the user's emotion of joy, and convert that scene into sign language or Braille."

[1048] In this way, the system provides information that responds to the user's emotions, allowing visually and hearing impaired people to enjoy watching sports more.

[1049] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1050] Step 1:

[1051] The device captures live audio data in real time using high-sensitivity microphones installed in the stadium. The input is the live audio data, and the output is the captured audio data. Specifically, the device uses a noise-canceling function to remove background noise.

[1052] Step 2:

[1053] The device transmits the captured voice data to the generative AI model on the cloud via the internet. The input is the captured voice data, and the output is the voice data sent to the generative AI model. Specifically, the voice data is compressed and sent to the server using a secure communication protocol.

[1054] Step 3:

[1055] The server monitors the user's connection to the system through a camera and microphone. The input is the user's facial expression data and tone of voice data, and the output is the data sent to the emotion engine. Specifically, the camera captures the user's facial expression, and the microphone collects the user's tone of voice.

[1056] Step 4:

[1057] The server uses an emotion engine to estimate the user's emotions from their facial expressions, tone of voice, and choice of words. The input is the user's facial expression data and tone of voice data, and the output is estimated emotion data. Specifically, the emotion engine analyzes the data in real time and estimates the user's emotions.

[1058] Step 5:

[1059] If the server estimates that the user's emotion is joy, it highlights exciting scenes in the live broadcast audio data. The input is the estimated emotion data and the live broadcast audio data, and the output is the data of the highlighted exciting scenes. Specifically, the server uses an algorithm to detect specific keywords (e.g., "goal" or "great play") in the live broadcast audio data.

[1060] Step 6:

[1061] The server converts the extracted exciting scenes into sign language or Braille. The input is the data of the highlighted exciting scenes, and the output is sign language video or Braille information. Specifically, the data is converted using sign language translation software or Braille conversion software.

[1062] Step 7:

[1063] The server sends the generated sign language video and Braille information to the user's device. The input is the sign language video and Braille information, and the output is the sign language video and Braille information displayed on the user's device. Specifically, a low-latency communication protocol is used to ensure real-time performance.

[1064] In this way, the system provides information that responds to the user's emotions, allowing visually and hearing impaired people to enjoy watching sports more.

[1065] (Application example 1)

[1066] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1067] When visually or hearing impaired users enjoy watching sports, simply converting live audio data into sign language or Braille in real time poses a challenge as it does not provide enough information that reflects the user's emotions.In addition, exciting scenes are not emphasized in a way that takes the user's emotions into consideration, making it difficult for visually or hearing impaired users to fully enjoy watching sports.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1069] In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, and means for emphasizing exciting scenes according to the emotions. This makes it possible to provide information according to the user's emotions so that visually or hearing impaired people can enjoy watching sports more.

[1070] "Live broadcast audio data" refers to audio information that conveys the progress of a sporting event or other live event in real time.

[1071] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate specific outputs.

[1072] "Sign language" is a means of communicating linguistic information using visual gestures and movements.

[1073] "Braille" is a pattern of raised dots on paper or other surfaces that can be read by the visually impaired using their sense of touch.

[1074] "Means for recognizing user emotions" refers to technology that estimates emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1075] "Means for emphasizing exciting scenes" refers to a technique for selecting particularly exciting scenes in a sports game and emphasizing those scenes for the visually or hearing impaired.

[1076] A system for implementing this invention is configured as follows: First, a server is equipped with a microphone and an audio input device for capturing live broadcast audio data. This audio input device can capture live broadcast audio data of a sporting event in real time.

[1077] The captured voice data is then sent to a generative AI, which analyzes the voice data and converts it into sign language or Braille in real time, which is then output in a format that can be understood by people with visual or hearing impairments.

[1078] Furthermore, the system is equipped with an emotion recognition engine that estimates the user's emotions from their facial expressions, tone of voice, choice of words, etc. The emotion recognition engine acquires and analyzes the user's emotional data using a camera and microphone.

[1079] Once the user's emotions are recognized, the system will highlight exciting scenes accordingly. For example, if the user expresses joy, the system will recognize that emotion and highlight particularly exciting scenes. This highlighted information can then be converted into sign language or Braille and provided to the visually or hearing impaired.

[1080] The specific hardware used is a microphone, camera, and server. The software uses Python, the speech_recognition library, the transformers library, and the opencv library. The combination of this hardware and software enables voice data capture, emotion recognition, conversion to sign language and braille, and highlighting exciting scenes.

[1081] For example, if a user expresses joy while watching a sports game, the system will recognize that emotion and convert it into sign language or Braille, highlighting particularly exciting scenes, and provide this information to the user in real time.

[1082] Examples of prompts to input to a generative AI model include:

[1083] If the user is showing the emotion of joy, how can you highlight the exciting scene and translate it into sign language or braille?

[1084] This prompt can be used to ask the generative AI model how to convert specific sign language or braille.

[1085] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1086] Step 1:

[1087] The server captures live broadcast audio data in real time using a microphone.

[1088] Input: Live broadcast audio data

[1089] Output: Captured audio data

[1090] Specific operation: The server captures the audio signal from the microphone as digital data and performs preprocessing for voice recognition.

[1091] Step 2:

[1092] The server sends the captured voice data to the generation AI for analysis.

[1093] Input: Captured audio data

[1094] Output: Analyzed audio data

[1095] Specific operation: The server inputs the voice data into the generative AI model, which performs speech recognition and natural language processing, converting the voice data into text data.

[1096] Step 3:

[1097] The server converts the analyzed voice data into sign language or Braille in real time.

[1098] Input: Parsed audio data

[1099] Output: Sign language data, Braille data

[1100] How it works: The server uses a generative AI model to run an algorithm that converts text data into sign language or Braille. The sign language data is output in a video format, and the Braille data is output in a format suitable for a tactile display.

[1101] Step 4:

[1102] The server uses a camera to capture the user's facial expressions and sends them to an emotion recognition engine.

[1103] Input: User's facial expression data

[1104] Output: Recognized emotion data

[1105] Specific operation: The server acquires video data from the camera and inputs it into the emotion recognition engine, which analyzes facial expressions and estimates the user's emotions.

[1106] Step 5:

[1107] The server highlights exciting scenes based on the recognized emotion data.

[1108] Input: Recognized emotion data, sign language data, Braille data

[1109] Output: Enhanced sign language data, Enhanced Braille data

[1110] Specific operation: The server analyzes the emotion data and selects and highlights particularly exciting scenes if the user expresses joy, thereby adjusting the sign language data and Braille data according to the user's emotion.

[1111] Step 6:

[1112] The server provides enhanced sign language and Braille data to the user.

[1113] Input: Enhanced sign language data, enhanced braille data

[1114] Output: Sign language data and Braille data provided to the user

[1115] Specific operation: The server sends the enhanced sign language data to the user in video format and the Braille data in a format suitable for tactile displays. The user receives this data in real time and can enjoy watching sports.

[1116] Example 2

[1117] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1118] When users with visual or hearing impairments enjoy watching sports, they face the challenge of having difficulty understanding the audio information of live broadcasts in real time.In addition, they face the problem of not being able to share the sense of realism and excitement of the event because the information provided is not tailored to the user's emotions.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1120] In this invention, the server includes means for capturing live broadcast audio data, means for preprocessing the live broadcast audio data, means for analyzing the preprocessed audio data using a generative AI model and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing a user's emotion from the audio data, and means for generating information based on the recognized emotion, thereby enabling users with visual or hearing impairments to enjoy watching sports in real time.

[1121] "Live broadcast audio data" is audio information provided in real time during live broadcasts of sports, events, etc.

[1122] "Preprocessing" refers to processes such as noise removal and normalization that are carried out to convert audio data into a format that is easier to analyze.

[1123] A "generative AI model" is an artificial intelligence model trained using deep learning techniques to analyze audio data and convert it into sign language or Braille.

[1124] "Sign language" is a visual, gestural language used by the blind to understand spoken information.

[1125] "Braille" is a writing system using raised dots that is used by visually impaired people to understand written information by touch.

[1126] "Emotion recognition" is an analytical technique for identifying a user's emotions from voice data.

[1127] "Information generation" is the process of creating information to provide to the user based on the analyzed voice data and recognized emotions.

[1128] "Output means" refers to a device or interface for providing the generated sign language or Braille information to the user.

[1129] The present invention is a system that enables visually or hearing impaired users to enjoy watching sports in real time. A specific embodiment of this system will be described below.

[1130] System configuration

[1131] Hardware

[1132] The server uses a microphone (e.g., a standard microphone) to capture voice data, has a high-performance processor and memory for processing and analyzing the voice data, and has a display or Braille display for outputting information in sign language or Braille.

[1133] software

[1134] The server uses the Python library Librosa to preprocess the audio data. The generative AI model is built using TensorFlow and PyTorch to analyze the audio data and convert it into sign language or Braille. OpenCV and the Emotion API are used for emotion recognition.

[1135] System Operation

[1136] The server captures live audio data from a microphone in real time. The captured audio data undergoes pre-processing such as noise reduction and normalization using Librosa. The pre-processed audio data is then fed into a generative AI model and converted into sign language or Braille.

[1137] At the same time, the server uses an emotion engine to recognize the user's emotions from the voice data in real time. Based on the recognized emotions, information tailored to the user's emotions is generated. For example, if the user expresses surprise, the system recognizes that emotion and converts it into sign language or Braille, emphasizing the surprise.

[1138] The generated sign language or Braille information is provided to the user via the device. In the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1139] Specific examples

[1140] When a user exclaims "Amazing!" while watching a sports game, the server captures the speech and inputs it into a generative AI model. The generative AI model analyzes the speech and generates data that converts the "Amazing!" into sign language or Braille. At the same time, the emotion engine recognizes the user's emotion of surprise and generates information that emphasizes the surprising scene according to that emotion.

[1141] Prompt Sentence Examples

[1142] "Describe a system that converts a user's voice of surprise while watching a sports game into sign language or Braille in real time, highlighting the surprising moment and providing information to the visually or hearing impaired."

[1143] This system allows users with visual or hearing impairments to enjoy watching sports in real time.

[1144] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1145] Step 1:

[1146] The server captures live audio data in real time through a microphone. The input is analog audio data, which is converted to digital format and stored in temporary storage. Specifically, the audio input device captures the audio and performs digital signal processing.

[1147] Step 2:

[1148] The server performs pre-processing on the captured audio data. The input is digital audio data, which is denoised and normalised using Librosa. The output is pre-processed audio data. Specifically, a noise filtering algorithm is applied and the volume level is equalised.

[1149] Step 3:

[1150] The server inputs preprocessed speech data into the generative AI model. The input is preprocessed speech data, which is analyzed by the generative AI model (using TensorFlow or PyTorch). The output is data for conversion into sign language or Braille. Specifically, the speech recognition algorithm analyzes the speech data and generates the corresponding sign language or Braille data.

[1151] Step 4:

[1152] The server uses an emotion engine to recognize user emotions from voice data. The input is preprocessed voice data, and emotions are identified using OpenCV and the Emotion API. The output is the recognized emotion data. Specifically, it analyzes the tone and intonation of the voice and identifies emotion categories (e.g., surprise, joy, sadness).

[1153] Step 5:

[1154] The server generates information based on the recognized emotion. The input is emotion data and sign language or Braille data, which are combined to generate information to provide to the user. The output is sign language or Braille information corresponding to the emotion. Specific operations include adjusting the content of the sign language or Braille based on the emotion data and setting points to emphasize.

[1155] Step 6:

[1156] The device then provides the generated sign language or Braille information to the user. The input is sign language or Braille information corresponding to the emotion, and the output is information displayed on a display or Braille display. Specifically, in the case of sign language, a sign language animation is displayed on the display, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1157] (Application example 2)

[1158] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1159] When visually or hearing impaired people enjoy watching sports, they have difficulty understanding live broadcast audio in real time. Furthermore, because information is not provided according to the user's emotions, there is a problem in that information cannot be emphasized in response to changes in emotions. This has led to the problem that visually or hearing impaired people cannot fully enjoy watching sports.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1161] In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing a user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and then enhancing the information and converting it into sign language or Braille. This allows visually and hearing impaired people to enjoy watching sports in real time. Furthermore, because information is emphasized according to the user's emotions, a more realistic viewing experience can be provided.

[1162] "Live broadcast audio data" is audio information obtained from live broadcasts of sporting events and other live events.

[1163] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[1164] "Sign language" is a visual, gestural language used by blind people to understand spoken information.

[1165] "Braille" is a pattern of raised dots used by the visually impaired to interpret written information by touch.

[1166] An "emotion engine" is a technology that recognizes a user's emotions in real time and generates information corresponding to those emotions.

[1167] "User emotion" refers to the emotional state, such as surprise, joy, or sadness, that a user experiences while watching a sporting event.

[1168] "Emphasis on information" means displaying specific information in a way that makes it stand out according to the user's emotions.

[1169] A system for carrying out this invention is configured as follows: The server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information in accordance with the emotions recognized by the emotion engine, and means for emphasizing the information and converting it into sign language or Braille.

[1170] Program processing explanation

[1171] The server captures live audio data from the smartphone's microphone. The captured audio data is converted to text using the speech_recognition library. Next, a generative AI model is used to convert the text data into sign language or Braille. Tensorflow is used to execute the sign language to Braille conversion model.

[1172] Furthermore, the emotion engine uses the emotion_recognition library to recognize the user's emotions in real time. Based on the recognized emotions, the generative AI emphasizes specific information and converts it into sign language or Braille, enabling it to provide information tailored to the user's emotions.

[1173] Specific examples

[1174] For example, if a user is watching a live sports broadcast, the audio commentary is picked up through the smartphone's microphone. The audio data is converted into text in real time, and the text is then converted into sign language or Braille. At the same time, an emotion engine recognizes the user's emotions and highlights specific scenes or information based on emotions such as surprise or joy.

[1175] Prompt Sentence Examples

[1176] "This application converts live audio commentary of sporting events into sign language and Braille in real time, providing information to help the visually and hearing impaired enjoy watching sports. It also uses an emotion engine to recognize the user's emotions and highlights information that corresponds to those emotions."

[1177] In this way, visually and hearing impaired people can enjoy watching sports in real time, and information is emphasized according to the user's emotions, providing a more realistic viewing experience.

[1178] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1179] Step 1:

[1180] The server acquires live broadcast audio data using the smartphone's microphone. The input is audio data from the smartphone's microphone, and the output is an audio data file. This audio data is used in subsequent processing steps.

[1181] Step 2:

[1182] The server converts the acquired voice data into text using the speech_recognition library. The input is the voice data acquired in step 1, and the output is text data. This conversion allows the voice information to be treated as text information.

[1183] Step 3:

[1184] The server converts the text data into sign language and Braille using a generative AI model. The input is the text data obtained in step 2, and the output is sign language data and Braille data. This conversion uses a sign language conversion model and a Braille conversion model using Tensorflow.

[1185] Step 4:

[1186] The server recognizes the user's emotions in real time using the emotion_recognition library. The input is the text data obtained in step 2, and the output is the user's emotional data. This emotional data is used for subsequent information enhancement processing.

[1187] Step 5:

[1188] The server emphasizes specific information and converts it into sign language or Braille based on the recognized emotion data. The input is the sign language data and Braille data obtained in step 3 and the emotion data obtained in step 4, and the output is the emphasized sign language data and Braille data. This processing makes it possible to provide information according to the user's emotions.

[1189] Step 6:

[1190] The server outputs the enhanced sign language data and Braille data to the user. The input is the enhanced sign language data and Braille data obtained in step 5, and the output is the sign language and Braille information displayed on the user's device. This allows visually and hearing impaired people to enjoy watching sports in real time.

[1191] Example 3

[1192] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1193] When users with visual or hearing impairments enjoy watching sports, it is difficult to provide them with real-time information that reflects their emotions. In particular, the lack of information that takes the user's emotions into consideration limits the user's experience.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1195] In this invention, the server includes means for acquiring user emotion data, means for analyzing the emotion data, means for generating sign language or Braille data based on the analyzed emotion data, and means for outputting the sign language or Braille data, thereby enabling real-time provision of information according to the user's emotions.

[1196] "User emotion data" is data that indicates the user's emotional state, obtained from the user's facial expressions, voice, etc.

[1197] "Means for acquiring" refers to devices and software for collecting user emotion data.

[1198] "Means for analyzing" refers to a device or software for analyzing the acquired emotional data and identifying the user's emotional state.

[1199] "Means for generating" refers to devices or software for generating sign language or Braille data based on analyzed emotional data.

[1200] "Output means" refers to devices or software that provide the generated sign language or Braille data to the user.

[1201] "Sign language data" is data that includes information expressed as sign language.

[1202] "Braille data" is data that includes information expressed as Braille.

[1203] "Real-time" refers to data acquisition, analysis, generation, and output occurring without delay.

[1204] "Blind and deaf" refers to people who are visually or hearing impaired.

[1205] MODE FOR CARRYING OUT THE INVENTION

[1206] This invention is a system that provides information in real time that corresponds to the emotions of visually or hearing impaired users so that they can enjoy watching sports. Specific embodiments of this system will be described below.

[1207] System configuration

[1208] 1. Acquiring user emotion data

[1209] The user inputs their facial expressions and voice using the device's camera and microphone. The device then transmits this data to the server in real time. The hardware used includes a camera and microphone.

[1210] 2. Emotion Data Analysis

[1211] The server passes the received facial expression and voice data to the emotion engine. The emotion engine analyzes the user's facial expression and tone of voice to identify the emotion the user is feeling. For example, if the user has a sad expression, the emotion engine will recognize the emotion as "sadness." The software used includes the emotion engine.

[1212] 3. Sign Language and Braille Data Generation

[1213] The server uses a generative AI model based on the emotion data obtained from the emotion engine to generate sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if a user expresses sadness, the generative AI model generates sign language and Braille data that emphasizes sad scenes.

[1214] 4. Data Output

[1215] The server sends the generated sign language data to the device's display and sends the Braille data to the Braille display. The device displays the sign language on the display and physically generates Braille on the Braille display. The user can receive information by looking at the display or touching the Braille display. The hardware used includes the display and the Braille display.

[1216] Specific examples

[1217] As a concrete example, consider the case where a user is watching a sports game. If the user expresses sadness, the emotion engine recognizes that emotion, and the server uses a generative AI model based on that information to generate sign language or Braille data that emphasizes the sad scene. The generated data is then provided to the user via a display or Braille display.

[1218] Prompt Sentence Examples

[1219] "If a user expresses sadness while watching a sports game, recognize that emotion and generate sign language or Braille data that highlights the sadness."

[1220] In this way, the server can provide information according to the user's emotions in real time, and can support the visually or hearing impaired to enjoy watching sports. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1221] Step 1:

[1222] The user inputs their facial expressions and voice through the device's camera and microphone. The device then transmits this data to the server in real time. The input is the user's facial expression data and voice data, and the output is emotion data sent to the server.

[1223] Step 2:

[1224] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotion the user is feeling. The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the emotion engine detects features such as "drooping eyebrows" and "drooping corners of the mouth" and determines that the user is expressing sadness.

[1225] Step 3:

[1226] The server uses a generative AI model based on the emotion data obtained from the emotion engine to generate sign language and Braille data. The input is analyzed emotion information, and the output is generated sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille data according to the emotion. For example, if a user expresses sadness, the generative AI model generates sign language and Braille data that emphasizes sad scenes.

[1227] Step 4:

[1228] The server sends the generated sign language data to the terminal's display and sends the Braille data to the Braille display. The input is the generated sign language data and Braille data, and the output is the sign language displayed on the display and Braille formed on the Braille display. The terminal displays the sign language on the display and physically forms Braille on the Braille display. The user can understand the sign language by looking at the display and read Braille by touching the Braille display.

[1229] In this way, the server can provide information in real time that corresponds to the user's emotions, helping visually or hearing impaired people to enjoy watching sports.

[1230] (Application example 3)

[1231] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1232] When users with visual or hearing impairments enjoy watching sports, they face the challenge of being unable to provide real-time information.In addition, the inability to provide information that responds to the user's emotions limits the user experience.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, means for generating sign language or Braille data according to the emotions, means for displaying the sign language data on a display, and means for outputting the Braille data to a Braille display. This enables users with visual or hearing impairments to enjoy watching sports in real time, and also makes it possible to provide information according to the user's emotions.

[1234] "Live broadcast audio data" refers to audio data that conveys the progress of a sporting event or other live event in real time.

[1235] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[1236] "Sign language" is a visual, gestural language used by deaf people to communicate.

[1237] "Braille" is a writing system using raised dots that is used by the visually impaired to read by touch.

[1238] A "display" is a device for visually displaying information.

[1239] A "Braille display" is a device that physically creates Braille characters that can be read by the visually impaired through touch.

[1240] "Means for recognizing user emotions" refers to technology that analyzes and recognizes emotions from the user's facial expressions, voice, etc.

[1241] "Means for generating sign language data" refers to technology that converts input information into sign language format.

[1242] "Means for generating Braille data" refers to technology that converts input information into Braille format.

[1243] "Means for displaying sign language data on a display" refers to technology for displaying the generated sign language data on a display.

[1244] "Means for outputting Braille data to a Braille display" refers to technology for outputting the generated Braille data to a Braille display.

[1245] A system for implementing the present invention has the following configuration.

[1246] System Program

[1247] The system includes a means for capturing live broadcast audio data, a means for analyzing the audio data using generation AI and converting it into sign language or Braille in real time, a means for outputting sign language or Braille, a means for recognizing the user's emotions, a means for generating sign language or Braille data according to the emotions, a means for displaying the sign language data on a display, and a means for outputting Braille data to a Braille display.

[1248] Processing Description

[1249] Hardware used

[1250] Smartphone camera

[1251] Smartphone displays

[1252] Braille display

[1253] Software used

[1254] OpenCV: Camera image capture and display

[1255] EmotionRecognizer: Emotion recognition model

[1256] SignLanguageGenerator: A sign language generation model

[1257] BrailleGenerator: A Braille Generation Model

[1258] Data processing and calculation

[1259] The server captures the user's face with the smartphone's camera and inputs the image into EmotionRecognizer to recognize the user's emotions. Based on the recognized emotions, SignLanguageGenerator generates sign language data and BrailleGenerator generates Braille data. The generated sign language data is displayed on the smartphone's display, and the Braille data is output to a Braille display.

[1260] Specific examples

[1261] For example, if a user expresses sadness while watching a sports game, the system will recognize that emotion, highlight the sadness, and convert it into sign language or Braille. A scene in which a player on a losing team is crying will be explained in detail in sign language or Braille.

[1262] Prompt Sentence Examples

[1263] Create a program that recognizes the user's emotions and generates sign language or Braille data according to the emotion. If the user expresses sadness, develop an application that provides information according to that emotion in sign language or Braille.

[1264] The above is an embodiment of the present invention. This system enables users with visual or hearing impairments to enjoy watching sports in real time, and also provides information that responds to the user's emotions.

[1265] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1266] Step 1:

[1267] The server captures the user's face using the smartphone camera. The input is the camera image, and the output is the captured image data. This image data is used as preprocessing for emotion recognition.

[1268] Step 2:

[1269] The server inputs the captured image data into EmotionRecognizer to recognize the user's emotions. The input is image data, and the output is recognized emotion data. EmotionRecognizer extracts facial features from the image data and classifies the emotion.

[1270] Step 3:

[1271] The server generates sign language data using SignLanguageGenerator based on the recognized emotion data. The input is emotion data and the output is sign language data. SignLanguageGenerator generates sign language actions corresponding to the emotion data.

[1272] Step 4:

[1273] The server generates Braille data using the BrailleGenerator based on the recognized emotion data. The input is emotion data and the output is Braille data. The BrailleGenerator generates Braille patterns corresponding to the emotion data.

[1274] Step 5:

[1275] The server displays the generated sign language data on the smartphone's display. The input is sign language data, and the output is sign language displayed on the display. The display provides the user with a visual representation of the sign language actions.

[1276] Step 6:

[1277] The server outputs the generated braille data to a braille display. The input is braille data, and the output is a braille display on the braille display. The braille display physically creates braille patterns that can be read by the user through touch.

[1278] Step 7:

[1279] When a user expresses emotion while watching a sporting event, the server recognizes that emotion in real time and generates and outputs appropriate sign language or Braille data, allowing the user to receive information according to their emotion in real time.

[1280] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1281] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1282] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1283] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1284] [Third embodiment]

[1285] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1286] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1287] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1288] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1289] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1290] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1291] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1292] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1293] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1294] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1295] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1296] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1297] "Example 1"

[1298] In one embodiment of the present invention, a microphone and audio input device are provided to capture live audio data from a sports broadcast. This audio input device can capture live audio data in real time. The captured audio data is then sent to the AI ​​generation system.

[1299] "Example 2"

[1300] The generative AI analyzes the captured voice data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the voice data. This generative AI is trained using AI technologies such as deep learning to achieve highly accurate conversion.

[1301] "Example 3"

[1302] The generated sign language or Braille data is provided to the user through an output device. Examples of output devices include a display for displaying sign language and a Braille display for outputting Braille. A specific example is a Braille display, which physically generates Braille based on the generated Braille data, allowing visually impaired people to read it by touch.

[1303] The processing flow of each embodiment will be described below.

[1304] "Example 1"

[1305] Step 1: Provide a microphone or audio input device that captures live broadcast audio data of a sports event. This audio input device can capture live broadcast audio data in real time.

[1306] Step 2: The captured voice data is sent to the generation AI, which is trained using AI techniques such as deep learning to achieve highly accurate conversion.

[1307] "Example 2"

[1308] Step 1: The generation AI analyzes the captured voice data and converts it into sign language or Braille in real time. Specifically, it generates sign language or Braille data based on information obtained from the voice data.

[1309] Step 2: The generated sign language or Braille data is provided to the user through an output device, such as a display for displaying sign language or a Braille display for outputting Braille.

[1310] "Example 3"

[1311] Step 1: As a concrete example, a Braille display physically forms Braille characters based on the generated Braille data, allowing visually impaired people to read them tactilely.

[1312] Example 1

[1313] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1314] People with visual or hearing impairments have difficulty enjoying watching sports. In particular, they have difficulty understanding audio information from live broadcasts in real time, which significantly reduces the enjoyment of watching sports.

[1315] The specification processing by the specification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for importing live broadcast audio data, a means for transmitting the live broadcast audio data to the generative AI model, and a means for the generative AI model to analyze the audio data and convert it into text. This enables people with visual or hearing impairments to enjoy watching sports in real time.

[1316] "Live broadcast audio data" is audio information provided in real time by a commentator at an event such as a sports game.

[1317] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates a specific output.

[1318] A "means" is a method or device used to achieve a particular purpose.

[1319] "Convert to text" is the process of analyzing audio data and converting it into text information.

[1320] "Output" refers to providing the processed data in a display, audio, or other format for presentation to a user.

[1321] MODE FOR CARRYING OUT THE INVENTION

[1322] This invention is a system that captures live audio data from sports broadcasts in real time and sends that data to a generative AI model. Specifically, the system acquires audio data using a microphone or audio input device, and sends the data to a generative AI model, where it performs various data processing and calculations.

[1323] Hardware and software used

[1324] Hardware: Microphone, audio input device

[1325] Software: Generative AI models (e.g., OpenAI's GPT-4)

[1326] System Overview

[1327] 1. Audio data capture:

[1328] A user broadcasts a live sports event.

[1329] A microphone or audio input device connected to the terminal captures audio data from the live broadcast in real time.

[1330] 2. Sending audio data:

[1331] The captured voice data is transmitted from the terminal to the server.

[1332] The server sends the received voice data to the generative AI model.

[1333] 3. Data analysis and processing:

[1334] The generative AI model analyzes the received voice data and performs the necessary data processing and calculations.

[1335] For example, it converts audio data into text, extracts important events, and generates real-time highlights.

[1336] 4. Result output:

[1337] The server receives the results from the generative AI model and provides feedback to the user.

[1338] Specifically, the generated text is sent to the user's terminal, and the terminal displays the text.

[1339] Specific examples

[1340] Example 1:

[1341] A user is broadcasting a live soccer match.

[1342] The microphone picks up the sound of "Goal!"

[1343] The terminal transmits this audio data to the server.

[1344] The server sends the voice data to the generative AI model, which then generates the text "A goal has been scored!"

[1345] Example 2:

[1346] A user is broadcasting a basketball game.

[1347] The microphone picks up the audio of "Three-point shot successful!"

[1348] The terminal transmits this audio data to the server.

[1349] The server sends the voice data to the generative AI model, which then generates the text "The three-point shot was successful!"

[1350] Prompt Sentence Examples

[1351] Prompt 1:

[1352] "Please analyze this audio data and extract the key events. Audio data: 'Goal!'"

[1353] Prompt Statement 2:

[1354] "Please convert this audio data to text. Audio data: 'Three-pointer made!'"

[1355] The above is an embodiment of the present invention.

[1356] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1357] Step 1:

[1358] A user is watching a sports game and broadcasting live commentary. A microphone and audio input device connected to the device captures the audio data of the live broadcast in real time. Specifically, when the user shouts "Goal!", the microphone captures the audio and converts it into digital data. The input is the user's voice, and the output is digital audio data.

[1359] Step 2:

[1360] The terminal sends the captured voice data to the server. Specifically, the terminal compresses the voice data and sends it to the server using the TCP / IP protocol. The input is digital voice data, and the output is the voice data sent to the server.

[1361] Step 3:

[1362] The server sends the received voice data to the generative AI model. Specifically, the server sends the voice data to the API of the generative AI model and waits for the analysis results. The input is the voice data sent to the server, and the output is the voice data sent to the generative AI model.

[1363] Step 4:

[1364] The generative AI model analyzes the voice data and performs the necessary data processing and calculations. Specifically, the generative AI model converts the voice data into text and generates the text "A goal has been scored!" The input is the voice data sent to the generative AI model, and the output is the generated text.

[1365] Step 5:

[1366] The server receives the results from the generative AI model and provides feedback to the user. Specifically, the server sends the generated text to the user's device, which then displays the text. The input is the generated text, and the output is the text displayed on the user's device.

[1367] (Application example 1)

[1368] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1369] In the past, when watching sports, it was difficult for people with visual or hearing impairments to understand the progress of the game and important highlights in real time. Furthermore, even for general spectators, there were limited means to obtain important highlights and statistical information of the game in real time. This limited the enjoyment of watching the game.

[1370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1371] In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the converted commentary, and means for outputting the generated highlights and statistical information. This enables visually or hearing impaired people to obtain information in real time to enjoy watching sports, and also enables general spectators to obtain important highlights and statistical information of the game in real time.

[1372] "Live broadcast audio data" refers to audio information provided in real time during live broadcasts of sports or events.

[1373] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate specific outputs.

[1374] "Sign language" is a language that uses hand and finger movements to communicate with people who are deaf or hard of hearing.

[1375] Braille is a tactile representation of letters made up of raised dots that is used by the visually impaired to read.

[1376] "Means for converting to text" refers to technology that analyzes voice data and converts it into corresponding text information.

[1377] "Highlights" refers to important moments or events in sports or events.

[1378] "Statistical Information" refers to numerical data and analytical results related to a match or event.

[1379] "Output means" refers to technology such as a display device or audio device for providing the generated information to the user.

[1380] A system for implementing this invention includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting the live broadcast audio data into text, means for generating important highlights and statistical information based on the textual commentary content, and means for outputting the generated highlights and statistical information.

[1381] Hardware and software used

[1382] Hardware: Smartphone (with built-in microphone)

[1383] Software: Python, SpeechRecognition library (voice recognition), OpenAI API (generative AI)

[1384] Processing flow

[1385] 1. Audio data capture:

[1386] The smartphone microphone is used to capture live audio data in real time. The user launches the smartphone application and starts voice input.

[1387] 2. Audio to text conversion:

[1388] The captured audio data is converted to text using the SpeechRecognition library. The server sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data.

[1389] 3. Generative AI analysis:

[1390] The transcribed commentary is sent to a generator AI using the OpenAI API, which analyzes the input text data and generates key highlights and statistics.

[1391] 4. Output of information:

[1392] The generated highlights and statistics are displayed in real time on the user's smartphone, allowing the user to keep track of key moments and statistics of the match in real time.

[1393] Specific examples

[1394] When a user is watching a soccer match, they launch a smartphone app and capture the audio from the live broadcast. The app converts the audio into text and displays important events such as "Goal!" and "Foul!" in real time. It also automatically generates and displays highlights and statistics from the match (e.g., number of shots, ball possession).

[1395] Prompt Sentence Examples

[1396] Generate highlights of the following sports plays:

[1397] "30 minutes into the first half, Team A scored a goal. Team B's defense collapsed. Team A continued to attack in the second half."

[1398] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1399] Step 1:

[1400] A user launches a smartphone application to capture the audio of a live broadcast. The user uses the smartphone's microphone to collect audio data in real time. The input is the audio data of the live broadcast, and the output is a stream of audio data.

[1401] Step 2:

[1402] The device converts the captured audio data into text using the SpeechRecognition library. The device sends the audio data to the Google Speech Recognition API and retrieves the corresponding text data. The input is a stream of audio data, and the output is text data.

[1403] Step 3:

[1404] The server sends the textual commentary to the generation AI using the OpenAI API. The server analyzes the input text data and generates important highlights and statistics. The input is text data, and the output is the generated highlights and statistics.

[1405] Step 4:

[1406] The server sends the generated highlights and statistical information to the user's smartphone in real time, where the user can view the information on the smartphone screen. The input is the generated highlights and statistical information, and the output is the information displayed on the smartphone screen.

[1407] Step 5:

[1408] Users can understand key moments and statistical information of a match in real time through a smartphone application. Specifically, users check the information on the screen and understand the progress of the match. The input is the information displayed on the smartphone screen, and the output is the user's understanding.

[1409] Example 2

[1410] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] People with visual or hearing impairments have difficulty understanding audio information in real time. In particular, the accuracy and speed of converting audio information into sign language or Braille is insufficient, which can delay the transmission of information or lead to misunderstandings. This makes it difficult for people with visual or hearing impairments to receive information accurately and quickly.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1413] In this invention, the server includes a means for capturing voice data, a means for converting the voice data into text data using voice recognition software, and a means for inputting the text data into a generative AI model to generate sign language data or Braille data, thereby enabling visually or hearing impaired people to receive voice information in sign language or Braille in real time.

[1414] "Audio data" refers to data in which audio is recorded in digital format.

[1415] "Speech recognition software" is software that analyzes voice data and converts it into text data.

[1416] "Text data" refers to data in which character information is recorded in digital format.

[1417] A "generative AI model" is an artificial intelligence model that is trained using deep learning technology and generates sign language data or Braille data based on text data.

[1418] "Sign language data" refers to data that records sign language movements in digital format.

[1419] "Braille data" refers to data that records Braille information in digital format.

[1420] "Sign language animation software" is software that visually displays sign language movements based on sign language data.

[1421] A "Braille display" is a device that physically displays Braille based on Braille data.

[1422] This invention is a system that enables people with visual or hearing impairments to receive audio information in real time as sign language or Braille. A specific embodiment of this system will be described below.

[1423] System configuration

[1424] Server Configuration

[1425] The server includes a means for capturing voice data, a means for converting the voice data into text data using voice recognition software, and a means for converting the text data into sign language data or Braille data using a generative AI model. The voice recognition software uses a general voice recognition API (e.g., Google Cloud Speech-to-Text API). The generative AI model is a model trained using deep learning technology (e.g., a model using TensorFlow or PyTorch).

[1426] Device configuration

[1427] The terminal includes a means for visually displaying the sign language data sent from the server using sign language animation software (e.g., SignAll), and a means for outputting Braille data to a Braille display (e.g., Focus Blue). The terminal also includes a microphone for the user to input voice data.

[1428] User operations

[1429] The user inputs voice data by speaking into the device's microphone. For example, they might say "hello." The device then sends this voice data to the server. The server converts the voice data into text data and generates sign language or Braille data using a generative AI model. The generated data is then sent to the device and displayed on the display as sign language animation or output to a Braille display.

[1430] Specific examples

[1431] Example 1: Conversion to sign language

[1432] When a user says "Good morning," the device sends the voice data to the server. The server converts the voice data into text and inputs the text information "Good morning" into the generative AI model. The generative AI model generates sign language data corresponding to "Good morning" and creates a sign language animation using sign language animation software. The device displays this sign language animation on the display, allowing the user to visually confirm the sign language for "Good morning."

[1433] Example 2: Conversion to Braille

[1434] When a user says "Thank you," the device sends the voice data to the server. The server converts the voice data into text and inputs the text information "Thank you" into the generative AI model. The generative AI model generates Braille data corresponding to "Thank you" and sends it to the Braille display. The Braille display connected to the device displays "Thank you" in Braille, allowing the user to read the Braille with their sense of touch.

[1435] Prompt Sentence Examples

[1436] "If a user says 'Good morning,' how would your generative AI model convert that into sign language or Braille?"

[1437] By inputting this prompt into a generative AI model, a detailed explanation of the specific conversion process can be obtained.

[1438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1439] Step 1:

[1440] The user inputs voice data.

[1441] The user speaks into the device's microphone. For example, they say "hello." The device captures this speech as digital audio data. The input is the user's voice, and the output is digital audio data.

[1442] Step 2:

[1443] The terminal transmits the voice data to the server.

[1444] The device transmits the captured audio data over the Internet to a server, where it is encoded in an appropriate format (e.g., WAV or MP3). The input is the digital audio data, and the output is the audio data transmitted to the server.

[1445] Step 3:

[1446] The server converts the voice data into text data.

[1447] The server converts the received voice data into text data using voice recognition software (for example, a general voice recognition API). For example, voice data saying "Hello" is converted into text data saying "Hello." The input is voice data, and the output is text data.

[1448] Step 4:

[1449] The server inputs the text data into the generative AI model.

[1450] The server inputs the converted text data into a generative AI model, which is trained using deep learning techniques. The input is text data, and the output is sign language data or Braille data.

[1451] Step 5:

[1452] A generative AI model generates sign language and Braille data.

[1453] A generative AI model generates sign language data or Braille data based on input text data. For example, the text data "Hello" is converted into sign language animation data or Braille data. The input is text data, and the output is sign language data or Braille data.

[1454] Step 6:

[1455] The server transmits the generated data to the terminal.

[1456] The server sends the generated sign language or braille data to the device. The data is encoded in an appropriate format (e.g., JSON or XML). The input is the sign language or braille data, and the output is the data sent to the device.

[1457] Step 7:

[1458] The device displays sign language animations and outputs braille data to a braille display.

[1459] The terminal displays the received sign language data on a display using sign language animation software (e.g., sign language animation software). In addition, the Braille data is output to a Braille display (e.g., a Braille display) connected to the terminal. The user can visually check the sign language animation displayed on the display and tactilely read the Braille displayed on the Braille display. The input is sign language data or Braille data, and the output is sign language animation or Braille.

[1460] (Application example 2)

[1461] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1462] People with visual or hearing impairments have difficulty understanding voice instructions and warnings in real time during security work, which can prevent them from working efficiently and lead to a decline in the quality of security.

[1463] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for converting audio data into text, means for converting the text into sign language and Braille, and means for providing the sign language and Braille to visually impaired and hearing impaired persons. This enables visually impaired and hearing impaired persons to understand audio instructions and warnings in real time and perform security operations efficiently.

[1464] "Live broadcast audio data" refers to audio information about events or situations occurring in real time.

[1465] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[1466] "Sign language" refers to a means of communicating linguistic information using visual gestures and movements.

[1467] "Braille" refers to a writing system that uses raised dots to enable visually impaired people to read written information using their sense of touch.

[1468] "Audio data" refers to information that records sound in digital form.

[1469] "Text" refers to textual information.

[1470] "Blind people" refers to people who are visually impaired.

[1471] "Deaf" refers to people who have hearing impairments.

[1472] "Security work" refers to work to protect the safety of facilities and people.

[1473] "Real-time" refers to processing and transmission occurring immediately without delay.

[1474] A system for carrying out the present invention converts voice data into sign language or Braille in real time and provides the converted data to visually impaired or hearing impaired people. A specific embodiment of the system will be described below.

[1475] System configuration

[1476] Hardware

[1477] Smartphone: Equipped with a microphone for capturing voice data and a display for showing the conversion results.

[1478] Server: Provides the computing resources for analyzing and converting audio data.

[1479] software

[1480] Speech Recognition Library: Use the speech_recognition library to convert audio data to text.

[1481] Sign Language Conversion Library: Use the hand_gesture_recognition library to convert text to sign language.

[1482] Braille conversion library: Use the braille_converter library to convert text to braille.

[1483] Generative AI models: Use deep learning models to analyze audio data and convert it into sign language or Braille.

[1484] System Operation

[1485] Acquiring audio data

[1486] The user inputs voice data using the smartphone's microphone. For example, a security guard may input "There is a suspicious person here."

[1487] Analysis of audio data

[1488] The server converts the captured audio data into text using the speech_recognition library, which serves as the basis for subsequent sign language and braille conversion.

[1489] Conversion to sign language and braille

[1490] The server uses generative AI models to convert text to sign language and braille using the hand_gesture_recognition library for sign language conversion and the braille_converter library for braille conversion.

[1491] Providing results

[1492] The converted sign language and Braille are then displayed on the smartphone's display, allowing visually and hearing impaired people to understand voice instructions and warnings in real time.

[1493] Specific examples

[1494] For example, if a security guard says "There's a suspicious person here," the system converts this speech into text and then into sign language and Braille, which are then displayed on the smartphone screen, making the information accessible to security guards who are visually or hearing impaired.

[1495] Prompt Sentence Examples

[1496] Enter the voice data. Example: "There is a suspicious person."

[1497] In this way, visually and hearing impaired people can understand voice instructions and warnings in real time and perform security tasks efficiently.

[1498] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1499] Step 1:

[1500] The user inputs voice data using the smartphone's microphone, which is temporarily stored in the smartphone's internal memory.

[1501] Step 2:

[1502] The smartphone sends the saved voice data to the server. The server receives the voice data and converts it into text using the speech_recognition library. The input is voice data and the output is text data.

[1503] Step 3:

[1504] The server analyzes the text data using a generative AI model. This analysis converts the text data into sign language and Braille. The input is text data, and the output is sign language and Braille data.

[1505] Step 4:

[1506] The server converts the sign language data into sign language using the hand_gesture_recognition library. The input is text data and the output is sign language data.

[1507] Step 5:

[1508] The server converts the braille data into braille using the braille_converter library. The input is text data and the output is braille data.

[1509] Step 6:

[1510] The server then sends the converted sign language and Braille data to the smartphone, which then displays the received data on its display, enabling visually and hearing impaired people to understand voice instructions and warnings in real time.

[1511] Example 3

[1512] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1513] The purpose of this system is to solve the problem that users with visual or hearing impairments have difficulty converting text data into sign language or Braille and using information in real time. In particular, there is a need for a system that can quickly and accurately convert text data entered by users into sign language or Braille and provide it through an appropriate output device.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1515] In this invention, the server includes a means for a user to input text data, a means for the server to receive the input data and send a prompt to the generative AI model, a means for the generative AI model to generate sign language or Braille data, a means for the server to send the generated data to an output device, and a means for the terminal to provide the sign language or Braille to the user through the output device, thereby enabling visually or hearing impaired users to quickly and accurately convert input text data into sign language or Braille and use information in real time.

[1516] "User" refers to a person who uses the system to input text data and receive the information converted into sign language or Braille.

[1517] "Server" refers to a computer system whose role is to receive input data from a user, send prompts to a generative AI model, and send the generated sign language or Braille data to an output device.

[1518] A "generative AI model" refers to an artificial intelligence model that generates sign language or Braille data based on text data from the user.

[1519] A "prompt sentence" refers to a sentence that instructs a generative AI model to convert text data into sign language or Braille.

[1520] An "output device" is a device for providing the generated sign language or Braille data to a user, and includes a sign language display, a Braille display, and the like.

[1521] "Sign language display" refers to a display device for visually displaying generated sign language data.

[1522] "Braille display" refers to a device that physically renders generated Braille data so that it can be read by the visually impaired through touch.

[1523] "Text data" refers to textual information that a user inputs into a system.

[1524] "Real-time" means that text data entered by the user is instantly converted into sign language or Braille and provided almost simultaneously through an output device.

[1525] MODE FOR CARRYING OUT THE INVENTION

[1526] This invention is a system that enables visually or hearing impaired users to convert text data into sign language or Braille and access information in real time. Specific embodiments of this system are described below.

[1527] System configuration

[1528] The system consists of the following major components:

[1529] 1. User terminal: A device that allows a user to input text data. Examples include PCs, smartphones, and tablets.

[1530] 2. Server: Receives input data from the user, sends prompts to the generative AI model, and sends the generated sign language or Braille data to the output device. Use a high-performance server (e.g., a server equipped with an NVIDIA GPU).

[1531] 3. Generative AI model: An artificial intelligence model for generating sign language or Braille data based on text data from users. It uses deep learning frameworks such as TensorFlow and PyTorch.

[1532] 4. Output device: A device that presents the generated sign language or Braille data to the user, including a sign language display or a Braille display.

[1533] System Operation

[1534] 1. The user enters text data.

[1535] A user inputs text data through the system's interface, for example, by typing "hello" using a web browser or a dedicated application.

[1536] 2. The server receives the input data and sends a prompt to the generative AI model.

[1537] The server receives the text data sent by the user. Based on the received data, it creates a prompt sentence and sends it to the generative AI model. For example, it sends the prompt sentence "Please convert the following text into sign language: Hello" to the generative AI model.

[1538] 3. The generative AI model generates sign language and Braille data.

[1539] The generative AI model analyzes the prompt received from the server and generates corresponding sign language or Braille data, such as sign language animation data or Braille sequence data.

[1540] 4. The server sends the generated data to an output device.

[1541] The server sends the sign language and Braille data received from the generative AI model to an output device, for example, sending sign language animation data to a sign language display and Braille data to a Braille display.

[1542] 5. The terminal provides sign language or Braille to the user through an output device.

[1543] The terminal receives the data sent from the server and provides it to the user through the output device. The sign language display shows the sign language animation, and the braille display physically generates braille. The user can see the sign language display visually and read the braille display tactilely.

[1544] Specific examples

[1545] For example, when the user inputs the text "Thank you," the specific operation is as follows.

[1546] 1. User: Enters "Thank you" into the input field in a web browser and clicks the send button.

[1547] 2. Server: Receives the text data "Thank you" from the user and sends the prompt sentence "Please convert the following text into sign language: Thank you" to the generative AI model.

[1548] 3. Generative AI model: Analyzes the prompt and generates sign language animation data corresponding to "thank you."

[1549] 4. Server: Sends the generated sign language animation data to the sign language display.

[1550] 5. Terminal: The sign language display shows the sign language animation, which the user can visually confirm.

[1551] In this way, the user can easily convert text into sign language or Braille and use it through an output device. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1552] Step 1:

[1553] The user inputs text data.

[1554] A user inputs text data using a web browser or a dedicated application. For example, the user inputs "Hello" and clicks the send button. The input data is sent from the user's device to the server.

[1555] Step 2:

[1556] The server receives the input data and sends prompts to the generative AI model.

[1557] The server receives text data sent by the user. Based on the received data, it creates a prompt sentence and sends it to the generative AI model. For example, the prompt sentence "Please convert the following text into sign language: Hello" is sent to the generative AI model. The input is the user's text data, and the output is the prompt sentence to the generative AI model.

[1558] Step 3:

[1559] A generative AI model generates sign language and Braille data.

[1560] The generative AI model analyzes the prompt received from the server and generates the corresponding sign language or Braille data. For example, it generates sign language animation data or Braille sequence data. The input is the prompt, and the output is the sign language or Braille data.

[1561] Step 4:

[1562] The server sends the generated data to an output device.

[1563] The server sends the sign language or Braille data received from the generative AI model to an output device. For example, it sends sign language animation data to a sign language display and Braille data to a Braille display. The input is the generated sign language or Braille data, and the output is the data sent to the output device.

[1564] Step 5:

[1565] The terminal provides sign language or Braille to the user through an output device.

[1566] The terminal receives data sent from the server and provides it to the user through the output device. The sign language display shows sign language animations, and the braille display physically produces braille. The user can see the sign language display visually and read the braille display tactilely. The input is sign language or braille data from the server, and the output is the provision of sign language or braille to the user.

[1567] (Application example 3)

[1568] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1569] People with visual or hearing impairments have difficulty receiving real-time information when watching sports or obtaining security information. While there is a need for rapid communication of information in sign language or Braille in emergencies, current systems are inadequate.

[1570] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing security camera footage, analyzing it using a sign language recognition model, and recognizing the sign language, means for displaying the recognized sign language on the footage, and means for outputting security information on a Braille display. This enables people with visual or hearing impairments to watch sports and receive security information in real time.

[1571] "Live broadcast audio data" refers to data that provides audio information about the progress of sporting events and other live events.

[1572] "Generative AI" is an artificial intelligence technology that analyzes input data and converts it into a specific format.

[1573] "Sign language" is a visual, gestural language used by deaf people to communicate.

[1574] "Braille" is a writing system using raised dots that is used by the visually impaired to read by touch.

[1575] A "sign language recognition model" is a machine learning model that analyzes sign language movements from camera footage and converts them into the corresponding language.

[1576] A "security camera" is a camera device for monitoring a specific area and acquiring video data.

[1577] A "Braille display" is a device that displays Braille characters using electronically controlled raised dots.

[1578] "Security information" is information relating to abnormalities or emergencies within the monitored area.

[1579] A system for implementing this invention is configured as follows: The server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for capturing security camera footage, analyzing it using a sign language recognition model, and recognizing the sign language, means for displaying the recognized sign language on the footage, and means for outputting security information on a Braille display.

[1580] Hardware and software used

[1581] Hardware:

[1582] Smartphone camera

[1583] Braille displays (e.g. BrailleDisplay)

[1584] security cameras

[1585] software:

[1586] OpenCV: Acquiring and displaying camera images

[1587] HandSignRecognition: A sign language recognition model

[1588] BrailleDisplay: A library for controlling braille displays

[1589] Data processing and calculation

[1590] 1. Importing live broadcast audio data:

[1591] The server captures play-by-play audio data that provides audio updates on the progress of sporting events and other live events.

[1592] 2. Generative AI analysis:

[1593] The server analyzes the captured live broadcast audio data using a generation AI and converts it into sign language or Braille in real time.

[1594] 3. Sign language and Braille output:

[1595] The server outputs the generated sign language or Braille through a display or Braille display.

[1596] 4. Security camera capture:

[1597] The server captures the security camera footage and analyzes it using a sign language recognition model.

[1598] 5. Sign Language Recognition and Display:

[1599] The server recognizes the sign language in the video using a sign language recognition model and displays the recognized sign language on the video.

[1600] 6. Outputting security information to a braille display:

[1601] The server outputs security information to a braille display.

[1602] Specific examples

[1603] Sports viewing:

[1604] Users with visual or hearing impairments can receive information from live broadcasts of sporting events by converting audio data into sign language or Braille in real time through a display or Braille display.

[1605] Obtaining security information:

[1606] The security camera captures the video, recognizes the sign language using a sign language recognition model, and displays the recognized sign language on the video. Security information is also output to a Braille display, allowing visually impaired people to receive the information tactilely.

[1607] Prompt Sentence Examples

[1608] "Create a program that analyzes security camera footage, displays emergency information in sign language, and outputs security information on a braille display."

[1609] The above is an embodiment of the present invention.

[1610] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1611] Step 1:

[1612] The server captures play-by-play audio data that provides audio information about the progress of a sporting event or other live event. The input is the play-by-play audio data, and the output is the captured audio data, which is used for subsequent analysis.

[1613] Step 2:

[1614] The server analyzes the captured live broadcast audio data using a generation AI and converts it into sign language or Braille in real time. The input is the live broadcast audio data, and the output is sign language data and Braille data. The generation AI analyzes the audio data and generates the corresponding sign language or Braille.

[1615] Step 3:

[1616] The server outputs the generated sign language and braille data through a display or braille display. The input is sign language data and braille data, and the output is sign language displayed on the display and braille displayed on the braille display. This allows users to receive information even if they are visually or hearing impaired.

[1617] Step 4:

[1618] The server captures the security camera footage and analyzes it using a sign language recognition model. The input is the security camera footage and the output is the recognized sign language data. The sign language recognition model analyzes the footage and recognizes the sign language movements.

[1619] Step 5:

[1620] The server displays the recognized sign language on the video. The input is the recognized sign language data, and the output is the sign language displayed on the video. This allows the user to understand the security camera video in sign language.

[1621] Step 6:

[1622] The server outputs security information to a braille display. The input is security information, and the output is a braille display on the braille display. This allows the user to receive security information tactilely.

[1623] The above are the specific processing steps for carrying out the present invention.

[1624] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1625] "Example 1"

[1626] One embodiment of the present invention is a system incorporating an emotion engine. This system recognizes a user's emotions and generates information based on those emotions to help visually or hearing-impaired people enjoy watching sports more. Specifically, the system infers emotions from the user's facial expressions, tone of voice, and word choice, and analyzes live broadcast audio data based on those emotions. For example, if a user expresses joy, the system recognizes that emotion and converts it into sign language or Braille, emphasizing the more exciting scenes.

[1627] "Example 2"

[1628] The emotion engine also recognizes emotions in real time and generates information based on those emotions. This makes it possible to provide information tailored to the user's real-time emotions, helping visually and hearing impaired people enjoy watching sports. For example, if a user expresses surprise, the system recognizes that emotion, highlights the surprise, and converts it into sign language or Braille.

[1629] "Example 3"

[1630] Furthermore, the emotion engine generates information based on the user's emotions and outputs that information in real time. This makes it possible to instantly provide information tailored to the user's emotions so that visually and hearing impaired people can enjoy watching sports. Specifically, if a user expresses sadness, the system recognizes that emotion, highlights sad scenes, and converts them into sign language or Braille.

[1631] The processing flow of each embodiment will be described below.

[1632] "Example 1"

[1633] Step 1: Estimate the user's emotions from their facial expressions, tone of voice, choice of words, etc.

[1634] Step 2: Analyze the live broadcast audio data according to the estimated emotions.

[1635] Step 3: Based on the analysis results, highlight the more exciting scenes and convert them into sign language or Braille.

[1636] "Example 2"

[1637] Step 1: Recognize user emotions in real time.

[1638] Step 2: Analyze the live broadcast audio data according to the recognized emotions.

[1639] Step 3: Based on the analysis results, highlight surprising scenes and convert them into sign language or Braille.

[1640] "Example 3"

[1641] Step 1: Recognize user emotions in real time and generate information based on those emotions.

[1642] Step 2: Output the generated information in real time.

[1643] Step 3: Based on the output information, highlight the sad scenes and convert them into sign language or Braille.

[1644] Example 1

[1645] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1646] When viewing sports, visually or hearing impaired users face the challenge of having difficulty understanding live audio information in real time. Another problem is that the enjoyment of watching a sporting event is not fully conveyed because information cannot be provided in response to the user's emotions.

[1647] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for importing live broadcast audio data, a means for analyzing the live broadcast audio data using a generative AI model and recognizing the user's emotion, and a means for converting the live broadcast audio data into sign language or Braille in real time according to the user's emotion. This makes it possible to provide information in real time so that users with visual or hearing impairments can enjoy watching sports.

[1648] "Live broadcast audio data" refers to data that provides real-time audio information about the progress of sporting events and other live events.

[1649] A "generative AI model" is an algorithm or system that uses artificial intelligence technology to analyze data and generate information for a specific purpose.

[1650] "Means for recognizing user emotions" refers to technology or devices that analyze a user's facial expressions, tone of voice, choice of words, etc. to estimate their emotional state.

[1651] "Means for converting into sign language or Braille" refers to software or hardware for converting audio data or text data into sign language video or Braille information.

[1652] "Means for outputting sign language or Braille" refers to an output device such as a display or Braille display that provides the generated sign language video or Braille information to the user.

[1653] An "exciting scene" is an important moment or play that creates particular excitement and emotion in watching sports.

[1654] This invention is a system that allows visually or hearing impaired users to enjoy watching sports. The system captures live broadcast audio data in real time, recognizes the user's emotions, and converts information corresponding to those emotions into sign language or Braille and provides it.

[1655] Hardware and software used

[1656] Hardware

[1657] Microphones: Installed in stadiums and event venues, they capture live audio data in real time.

[1658] Camera: Used to capture the user's facial expressions.

[1659] Terminal: A device that processes audio and video data and sends it to a server.

[1660] Braille display: An output device for providing braille information to a user.

[1661] software

[1662] Generative AI model: An algorithm that analyzes voice data and recognizes user emotions.

[1663] Emotion engine: Software that estimates emotions from a user's facial expressions, tone of voice, choice of words, etc.

[1664] Sign language translation software: Software for converting audio and text data into sign language video.

[1665] Braille conversion software: Software for converting audio and text data into Braille information.

[1666] Specific operation of the system

[1667] Importing audio data

[1668] The device uses high-sensitivity microphones installed in the stadium to capture live audio data in real time, for example, capturing the audio of the commentator shouting "Goal!"

[1669] Sending audio data

[1670] The device then transmits the captured audio data over the internet to a cloud-based generative AI model, where it is compressed and sent to a server using a secure communication protocol.

[1671] User emotion recognition

[1672] The server monitors the user's connection to the system via a camera and microphone. The emotion engine analyzes the user's facial expressions (e.g., smile or surprise) and tone of voice (e.g., excitement) in real time to estimate the user's emotions.

[1673] Emotion-based data analysis

[1674] If the server estimates that the user's emotion is joy, it will highlight exciting scenes from the live broadcast audio data, for example, by identifying and extracting the moments of goals and important plays.

[1675] Information generation and provision

[1676] The server converts the extracted exciting scenes into sign language and Braille. Specifically, it uses sign language translation software to generate sign language videos and Braille conversion software to display them on a Braille display. The generated information is sent to the user's device and displayed in real time.

[1677] Examples and prompts

[1678] Specific examples

[1679] If a user is watching a soccer match and shows a facial expression of joy when a goal is scored, the system will recognize that emotion and convert the commentary of the goal into sign language or Braille, highlighting it.

[1680] Prompt Sentence Examples

[1681] "Capture the audio of a live broadcast of the moment a goal is scored in a soccer match in real time, recognize the user's emotion of joy, and convert that scene into sign language or Braille."

[1682] In this way, the system provides information that responds to the user's emotions, allowing visually and hearing impaired people to enjoy watching sports more.

[1683] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1684] Step 1:

[1685] The device uses high-sensitivity microphones installed in the stadium to capture live audio data in real time. The input is the live audio data, and the output is the captured audio data. Specifically, the device uses a noise-canceling function to remove background noise.

[1686] Step 2:

[1687] The device transmits the captured voice data to the generative AI model on the cloud via the internet. The input is the captured voice data, and the output is the voice data sent to the generative AI model. Specifically, the voice data is compressed and sent to the server using a secure communication protocol.

[1688] Step 3:

[1689] The server monitors the user's connection to the system through a camera and microphone. The input is the user's facial expression data and tone of voice data, and the output is the data sent to the emotion engine. Specifically, the camera captures the user's facial expression, and the microphone collects the user's tone of voice.

[1690] Step 4:

[1691] The server uses an emotion engine to estimate the user's emotions from their facial expressions, tone of voice, and choice of words. The input is the user's facial expression data and tone of voice data, and the output is estimated emotion data. Specifically, the emotion engine analyzes the data in real time and estimates the user's emotions.

[1692] Step 5:

[1693] If the server estimates that the user's emotion is joy, it highlights exciting scenes in the live broadcast audio data. The input is the estimated emotion data and the live broadcast audio data, and the output is the data of the highlighted exciting scenes. Specifically, the server uses an algorithm to detect specific keywords (e.g., "goal" or "great play") in the live broadcast audio data.

[1694] Step 6:

[1695] The server converts the extracted exciting scenes into sign language or Braille. The input is the data of the highlighted exciting scenes, and the output is sign language video or Braille information. Specifically, the data is converted using sign language translation software or Braille conversion software.

[1696] Step 7:

[1697] The server sends the generated sign language video and Braille information to the user's device. The input is the sign language video and Braille information, and the output is the sign language video and Braille information displayed on the user's device. Specifically, a low-latency communication protocol is used to ensure real-time performance.

[1698] In this way, the system provides information that responds to the user's emotions, allowing visually and hearing impaired people to enjoy watching sports more.

[1699] (Application example 1)

[1700] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1701] When visually or hearing impaired users enjoy watching sports, simply converting live audio data into sign language or Braille in real time poses a challenge as it does not provide enough information that reflects the user's emotions.In addition, exciting scenes are not emphasized in a way that takes the user's emotions into consideration, making it difficult for visually or hearing impaired users to fully enjoy watching sports.

[1702] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1703] In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing the user's emotions, and means for emphasizing exciting scenes according to the emotions. This makes it possible to provide information according to the user's emotions so that visually or hearing impaired people can enjoy watching sports more.

[1704] "Live broadcast audio data" refers to audio information that conveys the progress of a sporting event or other live event in real time.

[1705] "Generative AI" is a system that uses artificial intelligence technology to analyze input data and generate specific outputs.

[1706] "Sign language" is a means of communicating linguistic information using visual gestures and movements.

[1707] "Braille" is a pattern of raised dots on paper or other surfaces that can be read by the visually impaired using their sense of touch.

[1708] "Means for recognizing user emotions" refers to technology that estimates emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1709] "Means for emphasizing exciting scenes" refers to a technique for selecting particularly exciting scenes in a sports game and emphasizing those scenes for the visually or hearing impaired.

[1710] A system for implementing this invention is configured as follows: First, a server is equipped with a microphone and an audio input device for capturing live broadcast audio data. This audio input device can capture live broadcast audio data of a sporting event in real time.

[1711] The captured voice data is then sent to a generative AI, which analyzes the voice data and converts it into sign language or Braille in real time, which is then output in a format that can be understood by people with visual or hearing impairments.

[1712] Furthermore, the system is equipped with an emotion recognition engine that estimates the user's emotions from their facial expressions, tone of voice, choice of words, etc. The emotion recognition engine acquires and analyzes the user's emotional data using a camera and microphone.

[1713] Once the user's emotions are recognized, the system will highlight exciting scenes accordingly. For example, if the user expresses joy, the system will recognize that emotion and highlight particularly exciting scenes. This highlighted information can then be converted into sign language or Braille and provided to the visually or hearing impaired.

[1714] The specific hardware used is a microphone, camera, and server. The software uses Python, the speech_recognition library, the transformers library, and the opencv library. The combination of this hardware and software enables voice data capture, emotion recognition, conversion to sign language and braille, and highlighting exciting scenes.

[1715] For example, if a user expresses joy while watching a sports game, the system will recognize that emotion and convert it into sign language or Braille, highlighting particularly exciting scenes, and provide this information to the user in real time.

[1716] Examples of prompts to input to a generative AI model include:

[1717] If the user is showing the emotion of joy, how can you highlight the exciting scene and translate it into sign language or braille?

[1718] This prompt can be used to ask the generative AI model how to convert specific sign language or braille.

[1719] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1720] Step 1:

[1721] The server captures live broadcast audio data in real time using a microphone.

[1722] Input: Live broadcast audio data

[1723] Output: Captured audio data

[1724] Specific operation: The server captures the audio signal from the microphone as digital data and performs preprocessing for voice recognition.

[1725] Step 2:

[1726] The server sends the captured voice data to the generation AI for analysis.

[1727] Input: Captured audio data

[1728] Output: Analyzed audio data

[1729] Specific operation: The server inputs the voice data into the generative AI model, which performs speech recognition and natural language processing, converting the voice data into text data.

[1730] Step 3:

[1731] The server converts the analyzed voice data into sign language or Braille in real time.

[1732] Input: Parsed audio data

[1733] Output: Sign language data, Braille data

[1734] How it works: The server uses a generative AI model to run an algorithm that converts text data into sign language or Braille. The sign language data is output in a video format, and the Braille data is output in a format suitable for a tactile display.

[1735] Step 4:

[1736] The server uses a camera to capture the user's facial expressions and sends them to an emotion recognition engine.

[1737] Input: User's facial expression data

[1738] Output: Recognized emotion data

[1739] Specific operation: The server acquires video data from the camera and inputs it into the emotion recognition engine, which analyzes facial expressions and estimates the user's emotions.

[1740] Step 5:

[1741] The server highlights exciting scenes based on the recognized emotion data.

[1742] Input: Recognized emotion data, sign language data, Braille data

[1743] Output: Enhanced sign language data, Enhanced Braille data

[1744] Specific operation: The server analyzes the emotion data and selects and highlights particularly exciting scenes if the user expresses joy, thereby adjusting the sign language data and Braille data according to the user's emotions.

[1745] Step 6:

[1746] The server provides enhanced sign language and Braille data to the user.

[1747] Input: Enhanced sign language data, enhanced braille data

[1748] Output: Sign language data and Braille data provided to the user

[1749] Specific operation: The server sends the enhanced sign language data to the user in video format and the Braille data in a format suitable for tactile displays. The user receives this data in real time and can enjoy watching sports.

[1750] Example 2

[1751] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1752] When users with visual or hearing impairments enjoy watching sports, they face the challenge of having difficulty understanding the audio information of live broadcasts in real time.In addition, they face the problem of not being able to share the sense of realism and excitement of the event because the information provided is not tailored to the user's emotions.

[1753] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1754] In this invention, the server includes means for capturing live broadcast audio data, means for preprocessing the live broadcast audio data, means for analyzing the preprocessed audio data using a generative AI model and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, means for recognizing a user's emotion from the audio data, and means for generating information based on the recognized emotion, thereby enabling users with visual or hearing impairments to enjoy watching sports in real time.

[1755] "Live broadcast audio data" is audio information provided in real time during live broadcasts of sports, events, etc.

[1756] "Preprocessing" refers to processes such as noise removal and normalization that are carried out to convert audio data into a format that is easier to analyze.

[1757] A "generative AI model" is an artificial intelligence model trained using deep learning techniques to analyze audio data and convert it into sign language or Braille.

[1758] "Sign language" is a visual, gestural language used by the blind to understand spoken information.

[1759] "Braille" is a writing system using raised dots that is used by visually impaired people to understand written information by touch.

[1760] "Emotion recognition" is an analytical technique for identifying a user's emotions from voice data.

[1761] "Information generation" is the process of creating information to provide to the user based on the analyzed voice data and recognized emotions.

[1762] "Output means" refers to a device or interface for providing the generated sign language or Braille information to the user.

[1763] The present invention is a system that enables visually or hearing impaired users to enjoy watching sports in real time. A specific embodiment of this system will be described below.

[1764] System configuration

[1765] Hardware

[1766] The server uses a microphone (e.g., a standard microphone) to capture voice data, has a high-performance processor and memory for processing and analyzing the voice data, and has a display or Braille display for outputting information in sign language or Braille.

[1767] software

[1768] The server uses the Python library Librosa to preprocess the audio data. The generative AI model is built using TensorFlow and PyTorch to analyze the audio data and convert it into sign language or Braille. OpenCV and the Emotion API are used for emotion recognition.

[1769] System Operation

[1770] The server captures live audio data from a microphone in real time. The captured audio data undergoes pre-processing such as noise reduction and normalization using Librosa. The pre-processed audio data is then input into a generative AI model and converted into sign language or Braille.

[1771] At the same time, the server uses an emotion engine to recognize the user's emotions from the voice data in real time. Based on the recognized emotions, information tailored to the user's emotions is generated. For example, if the user expresses surprise, the system recognizes that emotion, highlights the surprise, and converts it into sign language or Braille.

[1772] The generated sign language or Braille information is provided to the user via the device. In the case of sign language, a sign language animation is displayed on the screen, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1773] Specific examples

[1774] When a user exclaims "Amazing!" while watching a sports game, the server captures the speech and inputs it into a generative AI model. The generative AI model analyzes the speech and generates data that converts the "Amazing!" into sign language or Braille. At the same time, the emotion engine recognizes the user's emotion of surprise and generates information that emphasizes the surprising scene according to that emotion.

[1775] Prompt Sentence Examples

[1776] "Describe a system that converts a user's voice of surprise while watching a sports game into sign language or Braille in real time, highlighting the surprising moment and providing information to the visually or hearing impaired."

[1777] This system allows users with visual or hearing impairments to enjoy watching sports in real time.

[1778] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1779] Step 1:

[1780] The server captures live audio data in real time through a microphone. The input is analog audio data, which is converted to digital format and stored in temporary storage. Specifically, the audio input device captures the audio and performs digital signal processing.

[1781] Step 2:

[1782] The server performs pre-processing on the captured audio data. The input is digital audio data, which is denoised and normalised using Librosa. The output is pre-processed audio data. Specifically, a noise filtering algorithm is applied and the volume level is equalised.

[1783] Step 3:

[1784] The server inputs preprocessed speech data into the generative AI model. The input is preprocessed speech data, which is analyzed by the generative AI model (using TensorFlow or PyTorch). The output is data for conversion into sign language or Braille. Specifically, the speech recognition algorithm analyzes the speech data and generates the corresponding sign language or Braille data.

[1785] Step 4:

[1786] The server uses an emotion engine to recognize user emotions from voice data. The input is preprocessed voice data, and emotions are identified using OpenCV and the Emotion API. The output is the recognized emotion data. Specifically, it analyzes the tone and intonation of the voice and identifies emotion categories (e.g., surprise, joy, sadness).

[1787] Step 5:

[1788] The server generates information based on the recognized emotion. The input is emotion data and sign language or Braille data, which are combined to generate information to provide to the user. The output is sign language or Braille information corresponding to the emotion. Specific operations include adjusting the content of the sign language or Braille based on the emotion data and setting points to emphasize.

[1789] Step 6:

[1790] The device then provides the generated sign language or Braille information to the user. The input is sign language or Braille information corresponding to the emotion, and the output is information displayed on a display or Braille display. Specifically, in the case of sign language, a sign language animation is displayed on the display, and in the case of Braille, it is output to a Braille display. This allows users with visual or hearing impairments to enjoy watching sports in real time.

[1791] (Application example 2)

[1792] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1793] When visually or hearing impaired people enjoy watching sports, they have difficulty understanding live broadcast audio in real time. Furthermore, because information is not provided according to the user's emotions, there is a problem in that information cannot be emphasized in response to changes in emotions. This has led to the problem that visually or hearing impaired people cannot fully enjoy watching sports.

[1794] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1795] In this invention, the server includes means for capturing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing a user's emotions in real time, and means for generating information according to the emotions recognized by the emotion engine and then enhancing the information and converting it into sign language or Braille. This allows visually and hearing impaired people to enjoy watching sports in real time. Furthermore, because information is emphasized according to the user's emotions, a more realistic viewing experience can be provided.

[1796] "Live broadcast audio data" is audio information obtained from live broadcasts of sporting events and other live events.

[1797] "Generative AI" refers to artificial intelligence technology that analyzes voice data and converts it into sign language or Braille.

[1798] "Sign language" is a visual, gestural language used by blind people to understand spoken information.

[1799] "Braille" is a pattern of raised dots used by the visually impaired to interpret written information by touch.

[1800] An "emotion engine" is a technology that recognizes a user's emotions in real time and generates information according to those emotions.

[1801] "User emotion" refers to the emotional state, such as surprise, joy, or sadness, that a user experiences while watching a sporting event.

[1802] "Emphasis on information" means displaying specific information in a way that makes it stand out according to the user's emotions.

[1803] A system for carrying out this invention is configured as follows: The server includes means for importing live broadcast audio data, means for analyzing the live broadcast audio data using a generation AI and converting it into sign language or Braille in real time, means for outputting the sign language or Braille, an emotion engine for recognizing the user's emotions in real time, and means for generating information in accordance with the emotions recognized by the emotion engine, and means for emphasizing the information and converting it into sign language or Braille.

[1804] Program processing explanation

[1805] The server captures live audio data from the smartphone's microphone. The captured audio data is converted to text using the speech_recognition library. Next, a generative AI model is used to convert the text data into sign language or Braille. Tensorflow is used to execute the sign language to Braille conversion model.

[1806] Furthermore, the emotion engine uses the emotion_recognition library to recognize the user's emotions in real time. Based on the recognized emotions, the generative AI emphasizes specific information and converts it into sign language or Braille, enabling it to provide information tailored to the user's emotions.

[1807] Specific examples

[1808] For example, if a user is watching a live sports broadcast, the audio commentary is picked up through the smartphone's microphone. The audio data is converted into text in real time, and the text is then converted into sign language or Braille. At the same time, an emotion engine recognizes the user's emotions and highlights specific scenes or information based on emotions such as surprise or joy.

[1809] Prompt Sentence Examples

[1810] "This application converts live audio commentary of sporting events into sign language and Braille in real time, providing information to help the visually and hearing impaired enjoy watching sports. It also uses an emotion engine to recognize the user's emotions and highlights information that corresponds to those emotions."

[1811] In this way, visually and hearing impaired people can enjoy watching sports in real time, and information is emphasized according to the user's emotions, providing a more realistic viewing experience.

[1812] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1813] Step 1:

[1814] The server acquires live broadcast audio data using the smartphone's microphone. The input is audio data from the smartphone's microphone, and the output is an audio data file. This audio data is used in subsequent processing steps.

[1815] Step 2:

[1816] The server converts the acquired voice data into text using the speech_recognition library. The input is the voice data acquired in step 1, and the output is text data. This conversion allows the voice information to be treated as text information.

[1817] Step 3:

[1818] The server converts the text data into sign language and Braille using a generative AI model. The input is the text data obtained in step 2, and the output is sign language data and Braille data. This conversion uses a sign language conversion model and a Braille conversion model using Tensorflow.

[1819] Step 4:

[1820] The server recognizes the user's emotions in real time using the emotion_recognition library. The input is the text data obtained in step 2, and the output is the user's emotional data. This emotional data is used for subsequent information enhancement processing.

[1821] Step 5:

[1822] The server emphasizes specific information and converts it into sign language or Braille based on the recognized emotion data. The input is the sign language data and Braille data obtained in step 3 and the emotion data obtained in step 4, and the output is the emphasized sign language data and Braille data. This processing makes it possible to provide information according to the user's emotions.

[1823] Step 6:

[1824] The server outputs the enhanced sign language data and Braille data to the user. The input is the enhanced sign language data and Braille data obtained in step 5, and the output is the sign language and Braille information displayed on the user's device. This allows visually and hearing impaired people to enjoy watching sports in real time.

[1825] Example 3

[1826] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1827] When users with visual or hearing impairments enjoy watching sports, it is difficult to provide them with real-time information that reflects their emotions. In particular, the lack of information provided that takes the user's emotions into consideration limits the user's experience.

[1828] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1829] In this invention, the server includes means for acquiring user emotion data, means for analyzing the emotion data, means for generating sign language or Braille data based on the analyzed emotion data, and means for outputting the sign language or Braille data, thereby enabling real-time provision of information according to the user's emotions.

[1830] "User emotion data" is data that indicates the user's emotional state, obtained from the user's facial expressions, voice, etc.

[1831] "Means for acquiring" refers to devices and software for collecting user emotion data.

[1832] "Means for analyzing" refers to a device or software for analyzing the acquired emotional data and identifying the user's emotional state.

[1833] "Means for generating" refers to devices or software for generating sign language or Braille data based on analyzed emotional data.

[1834] "Output means" refers to devices or software that provide the generated sign language or Braille data to the user.

[1835] "Sign language data" is data that includes information expressed as sign language.

[1836] "Braille data" is data that includes information expressed as Braille.

[1837] "Real-time" refers to data acquisition, analysis, generation, and output occurring without delay.

[1838] "Blind and deaf" refers to people who are visually or hearing impaired.

[1839] MODE FOR CARRYING OUT THE INVENTION

[1840] This invention is a system that provides information in real time that corresponds to the emotions of visually or hearing impaired users so that they can enjoy watching sports. Specific embodiments of this system will be described below.

[1841] System configuration

[1842] 1. Acquiring user emotion data

[1843] The user inputs their facial expressions and voice using the device's camera and microphone. The device then transmits this data to the server in real time. The hardware used includes a camera and microphone.

[1844] 2. Emotion Data Analysis

[1845] The server passes the received facial expression and voice data to the emotion engine. The emotion engine analyzes the user's facial expression and tone of voice to identify the emotion the user is feeling. For example, if the user has a sad expression, the emotion engine will recognize the emotion as "sadness." The software used includes the emotion engine.

[1846] 3. Sign Language and Braille Data Generation

[1847] The server uses a generative AI model based on the emotion data obtained from the emotion engine to generate sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille according to the emotion. For example, if a user expresses sadness, the generative AI model generates sign language and Braille data that emphasizes sad scenes.

[1848] 4. Data Output

[1849] The server sends the generated sign language data to the device's display and sends the Braille data to the Braille display. The device displays the sign language on the display and physically generates Braille on the Braille display. The user can receive information by looking at the display or touching the Braille display. The hardware used includes the display and the Braille display.

[1850] Specific examples

[1851] As a concrete example, consider the case where a user is watching a sports game. If the user expresses sadness, the emotion engine recognizes that emotion, and the server uses a generative AI model based on that information to generate sign language or Braille data that emphasizes the sad scene. The generated data is then provided to the user via a display or Braille display.

[1852] Prompt Sentence Examples

[1853] "If a user expresses sadness while watching a sports game, recognize that emotion and generate sign language or Braille data that highlights the sadness."

[1854] In this way, the server can provide information according to the user's emotions in real time, and can support the visually or hearing impaired to enjoy watching sports. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1855] Step 1:

[1856] The user inputs their facial expressions and voice through the device's camera and microphone. The device then transmits this data to the server in real time. The input is the user's facial expression data and voice data, and the output is emotion data sent to the server.

[1857] Step 2:

[1858] The server passes the received facial and voice data to the emotion engine. The emotion engine analyzes the user's facial expressions and tone of voice to identify the emotion the user is feeling. The input is the emotion data sent to the server, and the output is the analyzed emotion information. Specifically, the emotion engine detects features such as "drooping eyebrows" and "drooping corners of the mouth" and determines that the user is expressing sadness.

[1859] Step 3:

[1860] The server uses a generative AI model based on the emotion data obtained from the emotion engine to generate sign language and Braille data. The input is analyzed emotion information, and the output is generated sign language and Braille data. The generative AI model has an algorithm that generates appropriate sign language and Braille data according to the emotion. For example, if a user expresses sadness, the generative AI model generates sign language and Braille data that emphasizes sad scenes.

[1861] Step 4:

[1862] The server sends the generated sign language data to the terminal's display and sends the Braille data to the Braille display. The input is the generated sign language data and Braille data, and the output is the sign language displayed on the display and Braille formed on the Braille display. The terminal displays the sign language on the display and physically forms Braille on the Braille display. The user can understand the sign language by looking at the display and read Braille by touching the Braille display.

[1863] In this way, the server can provide information in real time that corresponds to the user's emotions, helping visually or hearing impaired people to enjoy watching sports.

[1864] (Application example 3)

[1865] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1866] When users with visual or hearing impairments enjoy watching sports, they face the challenge of being unable to provide real-time information.In addition, the inability to provide information that responds to the user's emotions limits the user experience.

[1867] The specific processing by the specifi...

Claims

1. A means for acquiring audio information of a live broadcast of a sports event; means for generating a first prompt sentence for instructing conversion of the acquired voice information into text information; A means for generating text information corresponding to the acquired voice information by using the generated first prompt sentence and a generation AI model; means for generating a second prompt sentence for instructing conversion of the generated text information into sign language data or Braille data; A means for generating sign language data or Braille data corresponding to the generated text information using the generated second prompt sentence and the generative AI model; means for visually displaying the generated sign language data using sign language animation software; means for outputting the generated Braille data to a Braille display; A system including:

2. The method further includes a means for estimating the user's emotional state using an emotion engine based on the user's facial expression, tone of voice, choice of words, etc.; The means for generating sign language data or Braille data corresponding to the generated text information generates the sign language data or Braille data in which exciting scenes are emphasized according to the estimated emotional state of the user. The system of claim 1 .

3. A means for generating statistical information relating to important highlight scenes and game content based on the generated text information; A means for outputting the generated highlight footage and statistical information; The system of claim 1 further comprising:

Images