System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving audio data of a class or a lecture; means for converting the audio data into text data; means for receiving video data of a whiteboard or a blackboard; means for extracting character information from the video data; means for translating the text data and the character information into a desired language; means for generating a video based on the translated text data and the translated character information; and means for distributing the generated video.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Language barriers remain a major obstacle in modern education and lectures. Particularly in online classes and international conferences, communication between participants who speak different languages ​​can be difficult. There is a demand for technology that can translate not only speech but also content written on whiteboards and blackboards in real time. [Means for solving the problem]

[0005] The present invention is a system including means for receiving audio data of a class or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, and means for distributing the generated video. This system enables the content of a class or lecture to be translated in real time, thereby providing high-quality education and information that transcends language barriers.

[0006] "Audio data" is information that represents audio from classes, lectures, etc. in digital format.

[0007] "Text data" refers to information in which character information extracted from audio data or video data is expressed in digital form.

[0008] "Video data" refers to image or video information that digitally represents what is written on a whiteboard or blackboard.

[0009] A "whiteboard or blackboard" is a writing board used to visually present information in a lesson or lecture.

[0010] "Translation" is the act of converting text data expressed in one language into another language.

[0011] "Means" is a general term that refers to a device, process, method, or system used to accomplish a particular function or purpose.

[0012] Optical character recognition (OCR) is a technology that recognizes characters from images and videos and converts them into digital text.

[0013] "Real time" is a time concept in which data is received, processed, and output almost simultaneously.

[0014] "Distribution" is the act of sending created video and audio to viewers via the Internet or other means.

[0015] "Generation" is the act of creating new information or data based on multiple pieces of information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] System Overview

[0038] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. The system is composed of the following main components:

[0039] Voice Recognition Module

[0040] Character Recognition Module

[0041] Translation Module

[0042] Video generation and distribution module

[0043] Program processing

[0044] Speech Recognition Module:

[0045] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The captured audio data is converted into a format that is easy for the device to process (e.g., WAV, MP3) and sent to the server. The server passes the received audio data to a speech recognition module, which converts the audio data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[0046] Character Recognition Module:

[0047] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[0048] Translation Module:

[0049] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. The speech recognition "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy", while the character recognition "E = mc²" remains unchanged.

[0050] Video Creation and Distribution Module:

[0051] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles, allowing viewers to understand classes and lectures in their own language. The server distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen.

[0052] Specific examples

[0053] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[0054] This system allows users who speak different languages ​​to understand the same lessons or lectures in real time, making it possible to provide high-quality education and lecture content around the world.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[0058] Step 2:

[0059] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice data "Today we will talk about the law of conservation of energy" is converted into text data "Today we will talk about the law of conservation of energy."

[0060] Step 3:

[0061] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[0062] Step 4:

[0063] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data: "E = mc²."

[0064] Step 5:

[0065] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, the text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[0066] Step 6:

[0067] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[0068] Step 7:

[0069] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] Online classes and lectures can be difficult for participants to understand due to language differences. It is particularly important to provide high-quality translations in real time to accurately convey the content to audiences who speak different languages, but this is difficult with current technology.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes means for receiving audio data of a class or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, and means for distributing the generated video in real time, thereby enabling viewers who speak different languages ​​to understand the content of the class or lecture in real time.

[0075] "Audio data" refers to electronically configured audio signals obtained from classes, lectures, etc.

[0076] "Text data" is an electronic representation of character information extracted from audio data or video data.

[0077] "Whiteboard or blackboard video data" refers to electronically constructed visual information captured by a camera of what was written during class.

[0078] "Text information" refers to text extracted from video data, and includes mathematical expressions, words, and the like.

[0079] "Real-time" refers to processing or operations that occur at nearly instantaneous speeds.

[0080] "Optical character recognition technology" refers to technology that identifies and reads character information from an image.

[0081] "Subtitle information" refers to translated text data and text that overlays character information on video.

[0082] The "HTTP Live Streaming Protocol" is a communications standard for delivering media content in real time over the Internet.

[0083] "Server" refers to a computer system that processes, translates, and distributes audio and video data.

[0084] "Terminal" refers to an electronic device for capturing and playing audio and video.

[0085] "User" refers to the lecturer who gives the class or lecture, and the participants who watch and listen to it.

[0086] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. Specific embodiments of this system are described in detail below.

[0087] The system consists of three main components: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules are implemented using standard computers and network equipment.

[0088] Speech Recognition Module:

[0089] When a user starts an online class or lecture, the device uses a microphone to capture the user's voice data in real time. The captured voice data is converted by the device into an easy-to-process format such as WAV format and sent to the server. The server then uses a voice recognition module such as Google Cloud Speech-to-Text to convert the voice data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the server converts it into text as "Today I will talk about the law of conservation of energy."

[0090] Character Recognition Module:

[0091] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then uses a character recognition module such as Tesseract OCR to extract text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[0092] Translation Module:

[0093] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[0094] Video Creation and Distribution Module:

[0095] The server uses video generation software such as FFmpeg to generate new video based on the translated text data. The translated text data is overlaid on the generated video as subtitles. This method allows viewers to understand classes and lectures in their own language. The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The viewer's device receives the distributed video and displays it on the user's screen.

[0096] Specific examples

[0097] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to a server. The server uses Google Cloud Speech-to-Text to convert the voice data into text and Tesseract OCR to recognize the mathematical formulas and words written on the blackboard. This data is then translated into English using the Google Translate API, and a video with English subtitles is then generated using FFmpeg and distributed in real time to students overseas. Overseas students can then take the class with English subtitles on their own devices.

[0098] Prompt Sentence Examples

[0099] "Design a program for a system that generates and distributes video with English subtitles in real time based on Japanese audio data."

[0100] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0101] Processing Step Description

[0102] Step 1:

[0103] When a user starts an online class or lecture, the device connects to the system and uses a microphone to capture audio data in real time. The input is the user's voice, and the output is the captured audio data. Specifically, the device launches the audio capture software and collects the lecturer's words as audio data.

[0104] Step 2:

[0105] The device converts the captured audio data to WAV format and sends it to the server using the HTTP protocol. The input is the captured audio data and the output is WAV format audio data. Specifically, the device uses an audio format conversion library to convert the audio data to WAV format and a network library to send the data to the server.

[0106] Step 3:

[0107] The server uses a speech recognition module such as Google Cloud Speech-to-Text to convert the received WAV-formatted audio data into text data. The input is WAV-formatted audio data, and the output is text data. Specifically, the server calls a speech recognition API to convert the audio data into text.

[0108] Step 4:

[0109] While a user is writing on a whiteboard or blackboard, the device's camera captures video data and sends it to the server. The input is the video data of the whiteboard or blackboard, and the output is the captured video data. Specifically, the device launches video capture software, collects video data as frames, converts them into the target format (e.g., H.264), and sends them to the server.

[0110] Step 5:

[0111] The server uses a character recognition module such as Tesseract OCR to extract text information from the received video data. The input is the video data, and the output is the extracted text information. Specifically, the server runs OCR software to analyze the video data and extract the text information as text data.

[0112] Step 6:

[0113] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API to translate it into the desired language. The input is text data, and the output is translated text data. Specifically, the server calls the translation API to convert the text data into the specified language.

[0114] Step 7:

[0115] The server uses video generation software such as FFmpeg to generate a new video with the translated text data overlaid as subtitles. The input is the translated text data and the original video data, and the output is a new video with the subtitle information overlaid. Specifically, the server uses video editing software to overlay the text data onto the video and generate a new video file.

[0116] Step 8:

[0117] The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The input is the subtitled video, and the output is the video delivered to the viewer's device. Specifically, the server uses streaming server software to distribute the video file in live streaming format, and the viewer's device receives and plays the video.

[0118] (Application example 1)

[0119] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0120] In conventional online classes and lectures, it has been difficult for viewers who speak different languages ​​to understand the same content at the same time. Another problem is that real-time translation and subtitle generation cannot be performed with sufficient accuracy. Furthermore, there has been a lack of appropriate systems for delivering video in a format appropriate for the device used by the viewer.

[0121] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0122] In this invention, the server includes a means for receiving audio data of a lecture or class in real time and converting it into text data, a means for extracting video data from a whiteboard or blackboard using optical character recognition technology, a means for translating the text data and the character information into a desired language, a means for generating a video based on the translated text data and the translated character information and distributing it to a user's device in real time, and a means for displaying the generated video on the user's device. This allows viewers who speak different languages ​​to simultaneously understand the same content. Furthermore, viewers can receive the appropriately distributed video in real time using devices such as smartphones or head-mounted displays.

[0123] A "class or lecture" is a teaching or informational session held at an educational institution, business, etc.

[0124] "Audio data" is digital information of sound captured by a sound capturing device such as a microphone.

[0125] "Text data" refers to character information converted from audio data or video data.

[0126] "Video data" refers to digital information of video captured by a video camera or other video capture device.

[0127] Optical character recognition (OCR) is a technology for extracting characters from images and converting handwritten or printed characters into digital text.

[0128] "Translation means" refers to the process and techniques for converting text data expressed in one language into another language.

[0129] "Means for generating images" refers to the processes and technologies for creating digital images based on translated text data or character information.

[0130] "User device" refers to an electronic device used by a user to receive and display information, such as a smartphone or head-mounted display.

[0131] "Real-time delivery means" refers to the process and technology for instantly transmitting the generated video to the user's terminal.

[0132] This invention is a system that receives audio and video data of classes and lectures in real time, translates it into multiple languages, and distributes it to viewers. The system consists of the following main modules: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules work in combination to enable users who speak different languages ​​to simultaneously understand the same content.

[0133] Voice Recognition Module

[0134] Hardware: Smartphone microphone, head-mounted display microphone

[0135] Software: Speech recognition API (e.g., Google Cloud Speech-to-Text API)

[0136] Processing details: Audio data from classes and lectures is captured in real time and converted into text using a speech recognition API. For example, if a lecturer says, "Today, I'll talk about the law of conservation of energy," it will be converted into text as, "Today, I'll talk about the law of conservation of energy."

[0137] Character Recognition Module

[0138] Hardware: Smartphone camera, head-mounted display camera

[0139] Software: Optical Character Recognition technology (OCR, e.g. Google Cloud Vision API)

[0140] What it does: Captures text written on a whiteboard or blackboard and uses OCR to extract text information, such as "E = mc²," and converts it into digital text.

[0141] Translation Module

[0142] Software: Translation API (e.g., Google Cloud Translation API)

[0143] Processing content: Translate text data acquired through speech recognition or character recognition into the desired language. For example, "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[0144] Video generation and distribution module

[0145] Software: Video generation software (e.g., FFmpeg), real-time streaming software (e.g., WebRTC)

[0146] Processing details: Generates video based on translated text data and overlays subtitles. The generated video is delivered to the user's device in real time and displayed on the user's smartphone or head-mounted display.

[0147] Specific examples

[0148] When a user takes an online class in Japan, their smartphone or head-mounted display captures the instructor's voice and the contents of the blackboard. The server converts the voice into text using a speech recognition module and recognizes the contents of the blackboard using a character recognition module. This information is then translated into English using a translation module, and a video with English subtitles is generated and distributed in real time using a video generation and distribution module. Students overseas can take classes with English subtitles on their smartphones or head-mounted displays.

[0149] Prompt Sentence Examples

[0150] Please create a program that recognizes the audio and blackboard text of a Japanese online lecture in real time, translates it into English, and distributes it as a video with subtitles. Specific requirements are as follows:

[0151] 1. Convert audio to text using the Google Cloud Speech-to-Text API.

[0152] 2. The characters on the blackboard are recognized using Google Cloud Vision API (OCR function).

[0153] 3. Translation uses Google Cloud Translation API.

[0154] 4. FFmpeg is used to generate video and WebRTC is used for real-time streaming.

[0155] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0156] Step 1:

[0157] The device uses a microphone to capture audio data from classes and lectures in real time.

[0158] Input: The instructor's voice

[0159] Data processing: Converting audio data into digital format (e.g. WAV format)

[0160] Output: Digital audio data

[0161] Step 2:

[0162] The terminal transmits the captured audio data to a server over the Internet.

[0163] Input: Digital audio data

[0164] Data Computing: Data Transfer via the Internet

[0165] Output: Audio data received on the server

[0166] Step 3:

[0167] The server calls a speech recognition API to convert the speech data into text data.

[0168] Input: Audio data received on the server

[0169] Data processing: Convert to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text API).

[0170] Output: Text data generated by speech recognition

[0171] Step 4:

[0172] The device uses a camera to capture the contents of the whiteboard or blackboard.

[0173] Input: What is written on a whiteboard or blackboard

[0174] Data processing: Converting video data into digital format (e.g., JPEG or MP4 format)

[0175] Output: Digital video data

[0176] Step 5:

[0177] The terminal transmits the captured video data to a server via the Internet.

[0178] Input: Digital video data

[0179] Data Computing: Data Transfer via the Internet

[0180] Output: Video data received on the server

[0181] Step 6:

[0182] The server extracts text information from the video data using optical character recognition (OCR) technology.

[0183] Input: Video data received on the server

[0184] Data processing: Convert to text data using OCR technology (e.g., Google Cloud Vision API)

[0185] Output: Character information extracted by OCR

[0186] Step 7:

[0187] The server translates the text data generated by the speech recognition and character recognition into a desired language using a translation API.

[0188] Input: Speech recognition text data, character recognition text data

[0189] Data processing: Translate into the desired language using a translation API (e.g., Google Cloud Translation API)

[0190] Output: Translated text data

[0191] Step 8:

[0192] The server generates a video using the translated text data and overlays the subtitles onto the video.

[0193] Input: Translated text data

[0194] Data processing: Generate subtitled video using video generation software (e.g. FFmpeg)

[0195] Output: Video data with generated subtitles

[0196] Step 9:

[0197] The server distributes the generated subtitled video to the user's terminal in real time.

[0198] Input: Generated video data with subtitles

[0199] Data calculation: Transmitted to the user's device using real-time delivery technology (e.g., WebRTC)

[0200] Output: Subtitled video played on the user's device

[0201] Step 10:

[0202] The user's terminal displays the subtitled video delivered in real time.

[0203] Input: Video data with subtitles delivered in real time

[0204] Data processing: decoding and displaying video data

[0205] Output: Video with translated subtitles displayed on the user's device screen

[0206] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0207] System Overview

[0208] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[0209] Voice Recognition Module

[0210] Character Recognition Module

[0211] Translation Module

[0212] Video generation and distribution module

[0213] Emotion Engine

[0214] Program processing

[0215] Speech Recognition Module:

[0216] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[0217] Character Recognition Module:

[0218] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[0219] Translation Module:

[0220] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[0221] Video Creation and Distribution Module:

[0222] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle with the text "Today we will discuss the law of energy conservation" could be added. The server then distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen. Viewers can then follow lessons and lectures in their own language.

[0223] Emotion Engine:

[0224] A camera installed on the user's device captures the user's facial expression. The captured video data is sent to the server, which then passes it to the emotion engine. The emotion engine analyzes the user's facial expression and recognizes the emotion. The recognized emotion is fed back to the translation module and reflected in the translation method and image generation. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[0225] Specific examples

[0226] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[0227] Furthermore, the system captures the user's facial expressions in real time and uses an emotion engine to analyze their emotions. For example, if the system determines that the user is confused, it will provide additional explanations. In this way, high-quality educational and lecture content can be delivered worldwide, regardless of language barriers, to deepen user understanding.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[0231] Step 2:

[0232] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice "Today we will talk about the law of conservation of energy" is analyzed and converted into text data "Today we will talk about the law of conservation of energy."

[0233] Step 3:

[0234] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard in real time. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[0235] Step 4:

[0236] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data "E = mc²."

[0237] Step 5:

[0238] A camera installed on the user's device captures the user's facial expression. The captured facial data is sent to the server in real time. The server passes the received facial data to the emotion engine, which then analyzes the user's emotions. For example, if the user has a confused expression, the emotion engine will recognize this.

[0239] Step 6:

[0240] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy." "E = mc²" is left as is.

[0241] Step 7:

[0242] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[0243] Step 8:

[0244] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[0245] Step 9:

[0246] The emotion engine analyzes the user's emotion data and feeds it back to the translation module. For example, if the user looks confused, the translation module changes to provide a translation in simpler terms. Based on this emotion feedback, the server regenerates the video and delivers it with additional information to deepen the user's understanding.

[0247] In this way, the system translates audio and video content in real time, and even analyzes and provides feedback on the user's emotions, thereby enabling high-quality education and information provision.

[0248] Example 2

[0249] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0250] In online classes and lectures, real-time translation of audio and video content is important for overcoming language barriers. However, conventional systems have had difficulty not only translating audio and video data, but also appropriately reflecting the user's level of understanding and emotions. Furthermore, there was no way to analyze the user's emotions in real time and adjust the system's behavior based on that, making it difficult to provide efficient and effective feedback.

[0251] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the voice data into text data, means for receiving video data, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for capturing a user's facial expression and analyzing the emotion, and means for adjusting the system operation based on the analyzed emotion. This makes it possible not only to translate voice and video data in real time, but also to analyze the user's emotion and provide appropriate feedback.

[0252] "Audio data" is data that represents an audio signal in digital form.

[0253] "Text data" is character string data extracted from audio data or video data.

[0254] "Video data" refers to video data captured by a device such as a camera.

[0255] "Text information" refers to information such as characters and symbols extracted from video data.

[0256] "Translation" is the process of converting text data expressed in one language into another language.

[0257] "Distribution" means transmitting the generated video to viewers in real time via a network.

[0258] "User's facial expression" refers to the facial expression of the user captured by the camera.

[0259] "Analyzing emotions" refers to a process of identifying the emotions felt by the user based on the user's facial expression data.

[0260] "Adjusting the system's behavior" means changing the system's functions and behavior based on the analyzed emotional information.

[0261] System Overview

[0262] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[0263] Voice Recognition Module

[0264] Character Recognition Module

[0265] Translation Module

[0266] Video generation and distribution module

[0267] Emotion Engine

[0268] Voice Recognition Module

[0269] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data is converted into a format such as WAV or MP3 and sent to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," it will be converted into text data.

[0270] Character Recognition Module

[0271] When a user is teaching a lesson while writing on a whiteboard or blackboard, the device's camera captures the video. The captured video data is also sent to the server, and the server passes the received video data to a character recognition module (OCR). The character recognition module extracts character information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[0272] Translation Module

[0273] The server passes the text data extracted from the speech recognition module and character recognition module to the translation module, which translates the text data into the specified language. For example, "Today we will talk about the law of conservation of energy" is translated into English as "Today we will discuss the law of conservation of energy." The mathematical formula "E = mc²" is kept as is.

[0274] Video generation and distribution module

[0275] The server generates a new video based on the translated text data. This new video is overlaid with subtitles of the translated text. The generated video is then distributed to viewers in real time. Viewers receive this video on their own devices and display it on their screens.

[0276] Emotion Engine

[0277] The user's device uses a built-in camera to capture the user's facial expressions. The captured video data is sent to a server, which then passes it to an emotion engine to analyze the user's facial expressions. The emotion engine recognizes the user's emotions and feeds that information back to the system. Based on this feedback, the system can reflect the information in translations and video generation, if necessary. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[0278] Specific examples

[0279] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to the server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by the translation module, and then the video generation and distribution module generates a video with English subtitles and distributes it in real time to overseas students. The overseas students can then take the class with English subtitles on their own devices. Furthermore, the system captures the user's facial expressions in real time and analyzes their emotions with an emotion engine. For example, if the system determines that the user is confused, it provides additional supplementary explanations. In this way, high-quality education and lecture content can be provided worldwide, transcending language barriers and deepening users' understanding.

[0280] Example prompts for generative AI models

[0281] Example prompt 1:

[0282] "Please translate the Japanese audio data into English and distribute it with English subtitles."

[0283] Example prompt 2:

[0284] "Please recognize the mathematical formulas written on the blackboard and add them directly to the English subtitles."

[0285] Example prompt 3:

[0286] "Analyze the user's facial expression and display additional explanatory text if confused."

[0287] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0288] Step 1:

[0289] Terminal

[0290] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data (input) is converted into a format such as WAV or MP3 (data processing) and then sent to the server (output).

[0291] Specific behavior:

[0292] When a user starts speaking, "Today I'm going to talk about the law of conservation of energy," the audio is captured by the device's microphone. The captured audio data is converted into WAV format and sent from the device to the server.

[0293] Step 2:

[0294] server

[0295] The server passes the voice data (input) received from the terminal to the voice recognition module. This voice recognition module converts the voice data into character string data (text data) (data conversion). The character string data (output) is stored within the server.

[0296] Specific behavior:

[0297] When the server inputs the received voice data into the voice recognition module, the voice data "Today we will talk about the law of conservation of energy" is converted into text "Today we will talk about the law of conservation of energy."

[0298] Step 3:

[0299] Terminal

[0300] When a user is teaching a class while writing on a whiteboard or blackboard, the device camera captures this video. The captured video data (input) is sent to the server (output).

[0301] Specific behavior:

[0302] When a user writes "E = mc²" on the blackboard, the device's camera captures the image and the image data is sent to the server.

[0303] Step 4:

[0304] server

[0305] The server passes the received video data (input) to a character recognition module (OCR). This character recognition module extracts character information from the video data (data extraction). The extracted character information (output) is stored within the server.

[0306] Specific behavior:

[0307] When the video data received by the server is input into the character recognition module, the character information "E = mc²" is extracted and saved as text data.

[0308] Step 5:

[0309] server

[0310] The server passes the text data (input) obtained from the speech recognition module and character recognition module to the translation module. The translation module translates this data into the desired language (data translation). The translated text data (output) is stored within the server.

[0311] Specific behavior:

[0312] The text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy", and the text data "E = mc²" is left as is.

[0313] Step 6:

[0314] server

[0315] The server uses a video generation and distribution module to generate new video based on the translated text data (input). This module overlays the translated text data as subtitles (data generation). The generated video (output) is distributed to viewers in real time.

[0316] Specific behavior:

[0317] The server generates a video with an overlay of the subtitle "Today we will discuss the law of conservation of energy" and distributes it to viewers in real time.

[0318] Step 7:

[0319] Terminal

[0320] The user's device captures the user's facial expressions using a built-in camera, and the captured video data (input) is sent to the server (output).

[0321] Specific behavior:

[0322] If the user makes a confused expression, the expression is captured by the device's camera and the video data is sent to the server.

[0323] Step 8:

[0324] server

[0325] The server passes the received video data (input) to the emotion engine, which analyzes the user's facial expressions and recognizes their emotions (data analysis). The analyzed emotions (output) are used to adjust the system's behavior.

[0326] Specific behavior:

[0327] When the server inputs the captured user's facial expressions into an emotion engine, the emotion of confusion is recognized, and the system responds by generating and delivering additional explanatory text.

[0328] (Application example 2)

[0329] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Conventional online classes and lecture systems have had problems with real-time multilingual translation and providing feedback based on the user's level of understanding. Specifically, it is difficult to ensure that students and audience members who speak different languages ​​can instantly understand the content of the lecture. They also lack the functionality to supplement the content of the lecture based on the user's facial expressions and emotions. This has led to issues such as a decline in the quality of the classes and lectures and a decrease in the user's learning effectiveness.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0332] In this invention, the server includes means for receiving audio data of a lesson or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for receiving user facial expression data and analyzing the emotions, and means for providing additional explanations based on the analyzed emotion data. This makes it possible for users who speak different languages ​​to understand the content of the lesson in real time, and furthermore, by providing feedback based on the user's emotions, it is possible to improve the quality of the lesson or lecture and increase the user's learning effectiveness.

[0333] "Means for receiving audio data" refers to a device or function for capturing the audio of a class or lecture, converting it into digital data, and transmitting it to a server.

[0334] The "means for converting voice data into text data" refers to a voice recognition technology for analyzing received voice data and extracting its contents as text information.

[0335] "Means for receiving video data" refers to a device or function that captures what is written on a whiteboard or blackboard using a camera or other device and transmits it to a server as digital data.

[0336] The "means for extracting character information from video data" is a means for analyzing captured video data, recognizing characters therein, and converting them into text data.

[0337] "Means for translating text data and character information into a desired language" refers to a translation technology for converting text data obtained by speech recognition or character recognition into a language desired by the user.

[0338] "Means for generating video based on translated text data and translated character information" refers to video generation technology for creating new video based on translated text data and adding subtitles, etc. to that video.

[0339] The "means for distributing the generated video" refers to a network and software for distributing the generated video data to users in real time.

[0340] The "means for receiving the user's facial expression data and analyzing the emotions" is a technology for capturing the user's facial image in real time, analyzing the emotions, and acquiring them as data.

[0341] The "means for providing additional explanations based on analyzed emotion data" is an automatic response system that provides supplementary explanations of the content of a class or lecture based on the analyzed emotion of the user.

[0342] This invention relates to a system that performs real-time translation and sentiment analysis for online classes and lectures. This system consists of the following steps and components:

[0343] First, the server has a means for receiving the audio data of the class or lecture. This includes devices and functions that capture audio using a microphone installed on the terminal, convert the audio data into a digital format, and send it to the server. The server then converts the received audio data into text data using speech recognition technology.

[0344] Next, the server is equipped with a means for receiving video data. This includes a device or function that captures what is written on the whiteboard or blackboard with a camera, converts the video data into a digital format, and transmits it to the server. The server then extracts text information from the received video data using optical character recognition (OCR) technology.

[0345] Furthermore, the server translates the text data obtained by speech recognition and character recognition into the desired language using multilingual translation technology, such as the Google Translate API.

[0346] The server generates a new video based on the translated text data and the translated character information. This video generation includes technology to overlay the text data as subtitles. The generated video is delivered to the user's device in real time.

[0347] Furthermore, the server is equipped with a means for receiving the user's facial expression data and analyzing the emotions. The device uses a camera to capture the user's face and transmits the facial expression data to the server. The server then uses an emotion recognition algorithm to analyze the user's emotions in real time. For example, facial recognition technology and deep learning models can be used for emotion recognition.

[0348] Based on the analyzed emotional data, the server provides appropriate feedback. For example, if it determines that the user is confused, it provides additional explanations via voice or text. This feedback deepens the user's understanding and improves the quality of lessons or lectures.

[0349] A specific example of this application is a language learning platform. Using this system, users can understand lectures from instructors in multiple languages ​​in real time. Furthermore, if users have questions, additional explanations are automatically provided, improving learning effectiveness. This system is implemented using technologies such as Google Translate API, OpenCV, SpeechRecognition, and Pyttsx3.

[0350] Example prompt sentence:

[0351] "Please translate what is being said in Japanese into English and display it as subtitles on the screen. Additionally, please analyze the user's facial expressions and provide additional explanations if they are confused."

[0352] In this way, we provide a system that can perform translation and emotion analysis in real time to support user understanding.

[0353] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0354] Step 1:

[0355] Capture audio data

[0356] The device captures the audio data of a class or lecture using a microphone. The input is the teacher's or lecturer's audio data, and the output is audio data in a digital format (e.g., WAV, MP3). This digital data is sent to a server.

[0357] Step 2:

[0358] Speech-to-text conversion

[0359] The server passes the received audio data to a speech recognition module. The input is digital audio data, and the output is the corresponding text data. This conversion is done using a speech recognition library such as SpeechRecognition.

[0360] Step 3:

[0361] Video data capture

[0362] The device captures video data of the whiteboard or blackboard with a camera. The input is what the teacher or lecturer has written, and the output is digital video data. This digital data is sent to the server.

[0363] Step 4:

[0364] Extracting text information from video

[0365] The server passes the received video data to a character recognition module. The input is digital video data, and the output is extracted text information. This conversion is performed using optical character recognition (OCR) technology.

[0366] Step 5:

[0367] Text data translation

[0368] The server passes the text data from the speech recognition module and the character information from the character recognition module to the translation module. The input is the text data and character information, and the output is the text data translated into the desired language. This translation is performed using multilingual translation technologies such as DeepL or Google Translate API.

[0369] Step 6:

[0370] Video generation

[0371] The server generates new videos based on the translated text data. The input is the translated text data and character information, and the output is video data with subtitles added. Video generation technology is used for this generation.

[0372] Step 7:

[0373] Video distribution

[0374] The server distributes the generated video data to viewers in real time. The input is the generated video data, and the output is the video displayed on the user's device.

[0375] Step 8:

[0376] Capturing facial expression data

[0377] The device captures the user's facial expression with a camera. The input is the user's facial image data, and the output is the digital data. This digital data is sent to the server.

[0378] Step 9:

[0379] Emotion Analysis

[0380] The server passes the received facial expression data to the emotion engine. The input is digital facial image data, and the output is analyzed emotion data. This analysis is performed using a deep learning model.

[0381] Step 10:

[0382] Providing feedback

[0383] The server generates additional explanations as needed based on the analyzed emotion data. The input is emotion data, and the output is additional feedback information (audio or text). For example, if the server determines that the user is confused, it generates supplementary explanations and provides them in audio or text.

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0385] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0387] [Second embodiment]

[0388] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0389] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0390] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0392] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0394] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0395] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0396] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0399] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0400] System Overview

[0401] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. The system is composed of the following main components:

[0402] Voice Recognition Module

[0403] Character Recognition Module

[0404] Translation Module

[0405] Video generation and distribution module

[0406] Program processing

[0407] Speech Recognition Module:

[0408] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The captured audio data is converted into a format that is easy for the device to process (e.g., WAV, MP3) and sent to the server. The server passes the received audio data to a speech recognition module, which converts the audio data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[0409] Character Recognition Module:

[0410] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[0411] Translation Module:

[0412] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. The speech recognition "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy", while the character recognition "E = mc²" remains unchanged.

[0413] Video Creation and Distribution Module:

[0414] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles, allowing viewers to understand classes and lectures in their own language. The server distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen.

[0415] Specific examples

[0416] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[0417] This system allows users who speak different languages ​​to understand the same lessons or lectures in real time, making it possible to provide high-quality education and lecture content around the world.

[0418] The processing flow will be explained below.

[0419] Step 1:

[0420] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[0421] Step 2:

[0422] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice data "Today we will talk about the law of conservation of energy" is converted into text data "Today we will talk about the law of conservation of energy."

[0423] Step 3:

[0424] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[0425] Step 4:

[0426] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data: "E = mc²."

[0427] Step 5:

[0428] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, the text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[0429] Step 6:

[0430] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[0431] Step 7:

[0432] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[0433] Example 1

[0434] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0435] Online classes and lectures can be difficult for participants to understand due to language differences. It is particularly important to provide high-quality translations in real time to accurately convey the content to audiences who speak different languages, but this is difficult with current technology.

[0436] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0437] In this invention, the server includes means for receiving audio data of a class or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, and means for distributing the generated video in real time, thereby enabling viewers who speak different languages ​​to understand the content of the class or lecture in real time.

[0438] "Audio data" refers to electronically configured audio signals obtained from classes, lectures, etc.

[0439] "Text data" is an electronic representation of character information extracted from audio data or video data.

[0440] "Whiteboard or blackboard video data" refers to electronically constructed visual information captured by a camera of what was written during class.

[0441] "Text information" refers to text extracted from video data, and includes mathematical expressions, words, and the like.

[0442] "Real-time" refers to processing or operations that occur at nearly instantaneous speeds.

[0443] "Optical character recognition technology" refers to technology that identifies and reads character information from an image.

[0444] "Subtitle information" refers to translated text data and text that overlays character information on video.

[0445] The "HTTP Live Streaming Protocol" is a communications standard for delivering media content in real time over the Internet.

[0446] "Server" refers to a computer system that processes, translates, and distributes audio and video data.

[0447] "Terminal" refers to an electronic device for capturing and playing audio and video.

[0448] "User" refers to the lecturer who gives the class or lecture, and the participants who watch and listen to it.

[0449] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. Specific embodiments of this system are described in detail below.

[0450] The system consists of three main components: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules are implemented using standard computers and network equipment.

[0451] Speech Recognition Module:

[0452] When a user starts an online class or lecture, the device uses a microphone to capture the user's voice data in real time. The captured voice data is converted by the device into an easy-to-process format such as WAV format and sent to the server. The server then uses a voice recognition module such as Google Cloud Speech-to-Text to convert the voice data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the server converts it into text as "Today I will talk about the law of conservation of energy."

[0453] Character Recognition Module:

[0454] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then uses a character recognition module such as Tesseract OCR to extract text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[0455] Translation Module:

[0456] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[0457] Video Creation and Distribution Module:

[0458] The server uses video generation software such as FFmpeg to generate new video based on the translated text data. The translated text data is overlaid on the generated video as subtitles. This method allows viewers to understand classes and lectures in their own language. The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The viewer's device receives the distributed video and displays it on the user's screen.

[0459] Specific examples

[0460] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to a server. The server uses Google Cloud Speech-to-Text to convert the voice data into text and Tesseract OCR to recognize the mathematical formulas and words written on the blackboard. This data is then translated into English using the Google Translate API, and a video with English subtitles is then generated using FFmpeg and distributed in real time to students overseas. Overseas students can then take the class with English subtitles on their own devices.

[0461] Prompt Sentence Examples

[0462] "Design a program for a system that generates and distributes video with English subtitles in real time based on Japanese audio data."

[0463] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0464] Processing Step Description

[0465] Step 1:

[0466] When a user starts an online class or lecture, the device connects to the system and uses a microphone to capture audio data in real time. The input is the user's voice, and the output is the captured audio data. Specifically, the device launches the audio capture software and collects the lecturer's words as audio data.

[0467] Step 2:

[0468] The device converts the captured audio data to WAV format and sends it to the server using the HTTP protocol. The input is the captured audio data and the output is WAV format audio data. Specifically, the device uses an audio format conversion library to convert the audio data to WAV format and a network library to send the data to the server.

[0469] Step 3:

[0470] The server uses a speech recognition module such as Google Cloud Speech-to-Text to convert the received WAV-formatted audio data into text data. The input is WAV-formatted audio data, and the output is text data. Specifically, the server calls a speech recognition API to convert the audio data into text.

[0471] Step 4:

[0472] While a user is writing on a whiteboard or blackboard, the device's camera captures video data and sends it to the server. The input is the video data of the whiteboard or blackboard, and the output is the captured video data. Specifically, the device launches video capture software, collects video data as frames, converts them into the target format (e.g., H.264), and sends them to the server.

[0473] Step 5:

[0474] The server uses a character recognition module such as Tesseract OCR to extract text information from the received video data. The input is the video data, and the output is the extracted text information. Specifically, the server runs OCR software to analyze the video data and extract the text information as text data.

[0475] Step 6:

[0476] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API to translate it into the desired language. The input is text data, and the output is translated text data. Specifically, the server calls the translation API to convert the text data into the specified language.

[0477] Step 7:

[0478] The server uses video generation software such as FFmpeg to generate a new video with the translated text data overlaid as subtitles. The input is the translated text data and the original video data, and the output is a new video with the subtitle information overlaid. Specifically, the server uses video editing software to overlay the text data onto the video and generate a new video file.

[0479] Step 8:

[0480] The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The input is the subtitled video, and the output is the video delivered to the viewer's device. Specifically, the server uses streaming server software to distribute the video file in live streaming format, and the viewer's device receives and plays the video.

[0481] (Application example 1)

[0482] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0483] In conventional online classes and lectures, it has been difficult for viewers who speak different languages ​​to understand the same content at the same time. Another problem is that real-time translation and subtitle generation cannot be performed with sufficient accuracy. Furthermore, there has been a lack of appropriate systems for delivering video in a format appropriate for the device used by the viewer.

[0484] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0485] In this invention, the server includes a means for receiving audio data of a lecture or class in real time and converting it into text data, a means for extracting video data from a whiteboard or blackboard using optical character recognition technology, a means for translating the text data and the character information into a desired language, a means for generating a video based on the translated text data and the translated character information and distributing it to a user's device in real time, and a means for displaying the generated video on the user's device. This allows viewers who speak different languages ​​to simultaneously understand the same content. Furthermore, viewers can receive the appropriately distributed video in real time using devices such as smartphones or head-mounted displays.

[0486] A "class or lecture" is a teaching or informational session held at an educational institution, business, etc.

[0487] "Audio data" is digital information of sound captured by a sound capturing device such as a microphone.

[0488] "Text data" refers to character information converted from audio data or video data.

[0489] "Video data" refers to digital information of video captured by a video camera or other video capture device.

[0490] Optical character recognition (OCR) is a technology for extracting characters from images and converting handwritten or printed characters into digital text.

[0491] "Translation means" refers to the process and techniques for converting text data expressed in one language into another language.

[0492] "Means for generating images" refers to the processes and technologies for creating digital images based on translated text data or character information.

[0493] "User device" refers to an electronic device used by a user to receive and display information, such as a smartphone or head-mounted display.

[0494] "Real-time delivery means" refers to the process and technology for instantly transmitting the generated video to the user's terminal.

[0495] This invention is a system that receives audio and video data of classes and lectures in real time, translates it into multiple languages, and distributes it to viewers. The system consists of the following main modules: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules work in combination to enable users who speak different languages ​​to simultaneously understand the same content.

[0496] Voice Recognition Module

[0497] Hardware: Smartphone microphone, head-mounted display microphone

[0498] Software: Speech recognition API (e.g., Google Cloud Speech-to-Text API)

[0499] Processing details: Audio data from classes and lectures is captured in real time and converted into text using a speech recognition API. For example, if a lecturer says, "Today, I'll talk about the law of conservation of energy," it will be converted into text as, "Today, I'll talk about the law of conservation of energy."

[0500] Character Recognition Module

[0501] Hardware: Smartphone camera, head-mounted display camera

[0502] Software: Optical Character Recognition technology (OCR, e.g. Google Cloud Vision API)

[0503] What it does: Captures text written on a whiteboard or blackboard and uses OCR to extract text information, such as "E = mc²," and converts it into digital text.

[0504] Translation Module

[0505] Software: Translation API (e.g., Google Cloud Translation API)

[0506] Processing content: Translate text data acquired through speech recognition or character recognition into the desired language. For example, "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[0507] Video generation and distribution module

[0508] Software: Video generation software (e.g., FFmpeg), real-time streaming software (e.g., WebRTC)

[0509] Processing details: Generates video based on translated text data and overlays subtitles. The generated video is delivered to the user's device in real time and displayed on the user's smartphone or head-mounted display.

[0510] Specific examples

[0511] When a user takes an online class in Japan, their smartphone or head-mounted display captures the instructor's voice and the contents of the blackboard. The server converts the voice into text using a speech recognition module and recognizes the contents of the blackboard using a character recognition module. This information is then translated into English using a translation module, and a video with English subtitles is generated and distributed in real time using a video generation and distribution module. Students overseas can take classes with English subtitles on their smartphones or head-mounted displays.

[0512] Prompt Sentence Examples

[0513] Please create a program that recognizes the audio and blackboard text of a Japanese online lecture in real time, translates it into English, and distributes it as a video with subtitles. Specific requirements are as follows:

[0514] 1. Convert audio to text using the Google Cloud Speech-to-Text API.

[0515] 2. The characters on the blackboard are recognized using Google Cloud Vision API (OCR function).

[0516] 3. Translation uses Google Cloud Translation API.

[0517] 4. FFmpeg is used to generate video and WebRTC is used for real-time streaming.

[0518] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0519] Step 1:

[0520] The device uses a microphone to capture audio data from classes and lectures in real time.

[0521] Input: The instructor's voice

[0522] Data processing: Converting audio data into digital format (e.g. WAV format)

[0523] Output: Digital audio data

[0524] Step 2:

[0525] The terminal transmits the captured audio data to a server over the Internet.

[0526] Input: Digital audio data

[0527] Data Computing: Data Transfer via the Internet

[0528] Output: Audio data received on the server

[0529] Step 3:

[0530] The server calls a speech recognition API to convert the speech data into text data.

[0531] Input: Audio data received on the server

[0532] Data processing: Convert to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text API).

[0533] Output: Text data generated by speech recognition

[0534] Step 4:

[0535] The device uses a camera to capture the contents of the whiteboard or blackboard.

[0536] Input: What is written on a whiteboard or blackboard

[0537] Data processing: Converting video data into digital format (e.g., JPEG or MP4 format)

[0538] Output: Digital video data

[0539] Step 5:

[0540] The terminal transmits the captured video data to a server via the Internet.

[0541] Input: Digital video data

[0542] Data Computing: Data Transfer via the Internet

[0543] Output: Video data received on the server

[0544] Step 6:

[0545] The server extracts text information from the video data using optical character recognition (OCR) technology.

[0546] Input: Video data received on the server

[0547] Data processing: Convert to text data using OCR technology (e.g., Google Cloud Vision API)

[0548] Output: Character information extracted by OCR

[0549] Step 7:

[0550] The server translates the text data generated by the speech recognition and character recognition into a desired language using a translation API.

[0551] Input: Speech recognition text data, character recognition text data

[0552] Data processing: Translate into the desired language using a translation API (e.g., Google Cloud Translation API)

[0553] Output: Translated text data

[0554] Step 8:

[0555] The server generates a video using the translated text data and overlays the subtitles onto the video.

[0556] Input: Translated text data

[0557] Data processing: Generate subtitled video using video generation software (e.g. FFmpeg)

[0558] Output: Video data with generated subtitles

[0559] Step 9:

[0560] The server distributes the generated subtitled video to the user's terminal in real time.

[0561] Input: Generated video data with subtitles

[0562] Data calculation: Transmitted to the user's device using real-time delivery technology (e.g., WebRTC)

[0563] Output: Subtitled video played on the user's device

[0564] Step 10:

[0565] The user's terminal displays the subtitled video delivered in real time.

[0566] Input: Video data with subtitles delivered in real time

[0567] Data processing: decoding and displaying video data

[0568] Output: Video with translated subtitles displayed on the user's device screen

[0569] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0570] System Overview

[0571] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[0572] Voice Recognition Module

[0573] Character Recognition Module

[0574] Translation Module

[0575] Video generation and distribution module

[0576] Emotion Engine

[0577] Program processing

[0578] Speech Recognition Module:

[0579] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[0580] Character Recognition Module:

[0581] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[0582] Translation Module:

[0583] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[0584] Video Creation and Distribution Module:

[0585] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle with the text "Today we will discuss the law of energy conservation" could be added. The server then distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen. Viewers can then follow lessons and lectures in their own language.

[0586] Emotion Engine:

[0587] A camera installed on the user's device captures the user's facial expression. The captured video data is sent to the server, which then passes it to the emotion engine. The emotion engine analyzes the user's facial expression and recognizes the emotion. The recognized emotion is fed back to the translation module and reflected in the translation method and image generation. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[0588] Specific examples

[0589] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[0590] Furthermore, the system captures the user's facial expressions in real time and uses an emotion engine to analyze their emotions. For example, if the system determines that the user is confused, it will provide additional explanations. In this way, high-quality educational and lecture content can be delivered worldwide, regardless of language barriers, to deepen user understanding.

[0591] The processing flow will be explained below.

[0592] Step 1:

[0593] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[0594] Step 2:

[0595] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice "Today we will talk about the law of conservation of energy" is analyzed and converted into text data "Today we will talk about the law of conservation of energy."

[0596] Step 3:

[0597] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard in real time. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[0598] Step 4:

[0599] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data "E = mc²."

[0600] Step 5:

[0601] A camera installed on the user's device captures the user's facial expression. The captured facial data is sent to the server in real time. The server passes the received facial data to the emotion engine, which then analyzes the user's emotions. For example, if the user has a confused expression, the emotion engine will recognize this.

[0602] Step 6:

[0603] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy." "E = mc²" is left as is.

[0604] Step 7:

[0605] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[0606] Step 8:

[0607] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[0608] Step 9:

[0609] The emotion engine analyzes the user's emotion data and feeds it back to the translation module. For example, if the user looks confused, the translation module changes to provide a translation in simpler terms. Based on this emotion feedback, the server regenerates the video and delivers it with additional information to deepen the user's understanding.

[0610] In this way, the system translates audio and video content in real time, and even analyzes and provides feedback on the user's emotions, thereby enabling high-quality education and information provision.

[0611] Example 2

[0612] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0613] In online classes and lectures, real-time translation of audio and video content is important for overcoming language barriers. However, conventional systems have had difficulty not only translating audio and video data, but also appropriately reflecting the user's level of understanding and emotions. Furthermore, there was no way to analyze the user's emotions in real time and adjust the system's behavior based on that, making it difficult to provide efficient and effective feedback.

[0614] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the voice data into text data, means for receiving video data, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for capturing a user's facial expression and analyzing the emotion, and means for adjusting the system operation based on the analyzed emotion. This makes it possible not only to translate voice and video data in real time, but also to analyze the user's emotion and provide appropriate feedback.

[0615] "Audio data" is data that represents an audio signal in digital form.

[0616] "Text data" is character string data extracted from audio data or video data.

[0617] "Video data" refers to video data captured by a device such as a camera.

[0618] "Text information" refers to information such as characters and symbols extracted from video data.

[0619] "Translation" is the process of converting text data expressed in one language into another language.

[0620] "Distribution" means transmitting the generated video to viewers in real time via a network.

[0621] "User's facial expression" refers to the facial expression of the user captured by the camera.

[0622] "Analyzing emotions" refers to a process of identifying the emotions felt by the user based on the user's facial expression data.

[0623] "Adjusting the system's behavior" means changing the system's functions and behavior based on the analyzed emotional information.

[0624] System Overview

[0625] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[0626] Voice Recognition Module

[0627] Character Recognition Module

[0628] Translation Module

[0629] Video generation and distribution module

[0630] Emotion Engine

[0631] Voice Recognition Module

[0632] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data is converted into a format such as WAV or MP3 and sent to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," it will be converted into text data.

[0633] Character Recognition Module

[0634] When a user is teaching a lesson while writing on a whiteboard or blackboard, the device's camera captures the video. The captured video data is also sent to the server, and the server passes the received video data to a character recognition module (OCR). The character recognition module extracts character information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[0635] Translation Module

[0636] The server passes the text data extracted from the speech recognition module and character recognition module to the translation module, which translates the text data into the specified language. For example, "Today we will talk about the law of conservation of energy" is translated into English as "Today we will discuss the law of conservation of energy." The mathematical formula "E = mc²" is kept as is.

[0637] Video generation and distribution module

[0638] The server generates a new video based on the translated text data. This new video is overlaid with subtitles of the translated text. The generated video is then distributed to viewers in real time. Viewers receive this video on their own devices and display it on their screens.

[0639] Emotion Engine

[0640] The user's device uses a built-in camera to capture the user's facial expressions. The captured video data is sent to a server, which then passes it to an emotion engine to analyze the user's facial expressions. The emotion engine recognizes the user's emotions and feeds that information back to the system. Based on this feedback, the system can reflect the information in translations and video generation, if necessary. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[0641] Specific examples

[0642] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to the server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by the translation module, and then the video generation and distribution module generates a video with English subtitles and distributes it in real time to overseas students. The overseas students can then take the class with English subtitles on their own devices. Furthermore, the system captures the user's facial expressions in real time and analyzes their emotions with an emotion engine. For example, if the system determines that the user is confused, it provides additional supplementary explanations. In this way, high-quality education and lecture content can be provided worldwide, transcending language barriers and deepening users' understanding.

[0643] Example prompts for generative AI models

[0644] Example prompt 1:

[0645] "Please translate the Japanese audio data into English and distribute it with English subtitles."

[0646] Example prompt 2:

[0647] "Please recognize the mathematical formulas written on the blackboard and add them directly to the English subtitles."

[0648] Example prompt 3:

[0649] "Analyze the user's facial expression and display additional explanatory text if confused."

[0650] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0651] Step 1:

[0652] Terminal

[0653] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data (input) is converted into a format such as WAV or MP3 (data processing) and then sent to the server (output).

[0654] Specific behavior:

[0655] When a user starts speaking, "Today I'm going to talk about the law of conservation of energy," the audio is captured by the device's microphone. The captured audio data is converted into WAV format and sent from the device to the server.

[0656] Step 2:

[0657] server

[0658] The server passes the voice data (input) received from the terminal to the voice recognition module. This voice recognition module converts the voice data into character string data (text data) (data conversion). The character string data (output) is stored within the server.

[0659] Specific behavior:

[0660] When the server inputs the received voice data into the voice recognition module, the voice data "Today we will talk about the law of conservation of energy" is converted into text "Today we will talk about the law of conservation of energy."

[0661] Step 3:

[0662] Terminal

[0663] When a user is teaching a class while writing on a whiteboard or blackboard, the device camera captures this video. The captured video data (input) is sent to the server (output).

[0664] Specific behavior:

[0665] When a user writes "E = mc²" on the blackboard, the device's camera captures the image and the image data is sent to the server.

[0666] Step 4:

[0667] server

[0668] The server passes the received video data (input) to a character recognition module (OCR). This character recognition module extracts character information from the video data (data extraction). The extracted character information (output) is stored within the server.

[0669] Specific behavior:

[0670] When the video data received by the server is input into the character recognition module, the character information "E = mc²" is extracted and saved as text data.

[0671] Step 5:

[0672] server

[0673] The server passes the text data (input) obtained from the speech recognition module and character recognition module to the translation module. The translation module translates this data into the desired language (data translation). The translated text data (output) is stored within the server.

[0674] Specific behavior:

[0675] The text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy", and the text data "E = mc²" is left as is.

[0676] Step 6:

[0677] server

[0678] The server uses a video generation and distribution module to generate new video based on the translated text data (input). This module overlays the translated text data as subtitles (data generation). The generated video (output) is distributed to viewers in real time.

[0679] Specific behavior:

[0680] The server generates a video with an overlay of the subtitle "Today we will discuss the law of conservation of energy" and distributes it to viewers in real time.

[0681] Step 7:

[0682] Terminal

[0683] The user's device captures the user's facial expressions using a built-in camera, and the captured video data (input) is sent to the server (output).

[0684] Specific behavior:

[0685] If the user makes a confused expression, the expression is captured by the device's camera and the video data is sent to the server.

[0686] Step 8:

[0687] server

[0688] The server passes the received video data (input) to the emotion engine, which analyzes the user's facial expressions and recognizes their emotions (data analysis). The analyzed emotions (output) are used to adjust the system's behavior.

[0689] Specific behavior:

[0690] When the server inputs the captured user's facial expressions into an emotion engine, the emotion of confusion is recognized, and the system responds by generating and delivering additional explanatory text.

[0691] (Application example 2)

[0692] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0693] Conventional online classes and lecture systems have had problems with real-time multilingual translation and providing feedback based on the user's level of understanding. Specifically, it is difficult to ensure that students and audience members who speak different languages ​​can instantly understand the content of the lecture. They also lack the functionality to supplement the content of the lecture based on the user's facial expressions and emotions. This has led to issues such as a decline in the quality of the classes and lectures and a decrease in the user's learning effectiveness.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0695] In this invention, the server includes means for receiving audio data of a lesson or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for receiving user facial expression data and analyzing the emotions, and means for providing additional explanations based on the analyzed emotion data. This makes it possible for users who speak different languages ​​to understand the content of the lesson in real time, and furthermore, by providing feedback based on the user's emotions, it is possible to improve the quality of the lesson or lecture and increase the user's learning effectiveness.

[0696] "Means for receiving audio data" refers to a device or function for capturing the audio of a class or lecture, converting it into digital data, and transmitting it to a server.

[0697] The "means for converting voice data into text data" refers to a voice recognition technology for analyzing received voice data and extracting its contents as text information.

[0698] "Means for receiving video data" refers to a device or function that captures what is written on a whiteboard or blackboard using a camera or other device and transmits it to a server as digital data.

[0699] The "means for extracting character information from video data" is a means for analyzing captured video data, recognizing characters therein, and converting them into text data.

[0700] "Means for translating text data and character information into a desired language" refers to a translation technology for converting text data obtained by speech recognition or character recognition into a language desired by the user.

[0701] "Means for generating video based on translated text data and translated character information" refers to video generation technology for creating new video based on translated text data and adding subtitles, etc. to that video.

[0702] The "means for distributing the generated video" refers to a network and software for distributing the generated video data to users in real time.

[0703] The "means for receiving the user's facial expression data and analyzing the emotions" is a technology for capturing the user's facial image in real time, analyzing the emotions, and acquiring them as data.

[0704] The "means for providing additional explanations based on analyzed emotion data" is an automatic response system that provides supplementary explanations of the content of a class or lecture based on the analyzed emotion of the user.

[0705] This invention relates to a system that performs real-time translation and sentiment analysis for online classes and lectures. This system consists of the following steps and components:

[0706] First, the server has a means for receiving the audio data of the class or lecture. This includes devices and functions that capture audio using a microphone installed on the terminal, convert the audio data into a digital format, and send it to the server. The server then converts the received audio data into text data using speech recognition technology.

[0707] Next, the server is equipped with a means for receiving video data. This includes a device or function that captures what is written on the whiteboard or blackboard with a camera, converts the video data into a digital format, and transmits it to the server. The server then extracts text information from the received video data using optical character recognition (OCR) technology.

[0708] Furthermore, the server translates the text data obtained by speech recognition and character recognition into the desired language using multilingual translation technology, such as the Google Translate API.

[0709] The server generates a new video based on the translated text data and the translated character information. This video generation includes technology to overlay the text data as subtitles. The generated video is delivered to the user's device in real time.

[0710] Furthermore, the server is equipped with a means for receiving the user's facial expression data and analyzing the emotions. The device uses a camera to capture the user's face and transmits the facial expression data to the server. The server then uses an emotion recognition algorithm to analyze the user's emotions in real time. For example, facial recognition technology and deep learning models can be used for emotion recognition.

[0711] Based on the analyzed emotional data, the server provides appropriate feedback. For example, if it determines that the user is confused, it provides additional explanations via voice or text. This feedback deepens the user's understanding and improves the quality of lessons or lectures.

[0712] A specific example of this application is a language learning platform. Using this system, users can understand lectures from instructors in multiple languages ​​in real time. Furthermore, if users have questions, additional explanations are automatically provided, improving learning effectiveness. This system is implemented using technologies such as Google Translate API, OpenCV, SpeechRecognition, and Pyttsx3.

[0713] Example prompt sentence:

[0714] "Please translate what is being said in Japanese into English and display it as subtitles on the screen. Additionally, please analyze the user's facial expressions and provide additional explanations if they are confused."

[0715] In this way, we provide a system that can perform translation and emotion analysis in real time to support user understanding.

[0716] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0717] Step 1:

[0718] Capture audio data

[0719] The device captures the audio data of a class or lecture using a microphone. The input is the teacher's or lecturer's audio data, and the output is audio data in a digital format (e.g., WAV, MP3). This digital data is sent to a server.

[0720] Step 2:

[0721] Speech-to-text conversion

[0722] The server passes the received audio data to a speech recognition module. The input is digital audio data, and the output is the corresponding text data. This conversion is done using a speech recognition library such as SpeechRecognition.

[0723] Step 3:

[0724] Video data capture

[0725] The device captures video data of the whiteboard or blackboard with a camera. The input is what the teacher or lecturer has written, and the output is digital video data. This digital data is sent to the server.

[0726] Step 4:

[0727] Extracting text information from video

[0728] The server passes the received video data to a character recognition module. The input is digital video data, and the output is extracted text information. This conversion is performed using optical character recognition (OCR) technology.

[0729] Step 5:

[0730] Text data translation

[0731] The server passes the text data from the speech recognition module and the character information from the character recognition module to the translation module. The input is the text data and character information, and the output is the text data translated into the desired language. This translation is performed using multilingual translation technologies such as DeepL or Google Translate API.

[0732] Step 6:

[0733] Video generation

[0734] The server generates new videos based on the translated text data. The input is the translated text data and character information, and the output is video data with subtitles added. Video generation technology is used for this generation.

[0735] Step 7:

[0736] Video distribution

[0737] The server distributes the generated video data to viewers in real time. The input is the generated video data, and the output is the video displayed on the user's device.

[0738] Step 8:

[0739] Capturing facial expression data

[0740] The device captures the user's facial expression with a camera. The input is the user's facial image data, and the output is the digital data. This digital data is sent to the server.

[0741] Step 9:

[0742] Emotion Analysis

[0743] The server passes the received facial expression data to the emotion engine. The input is digital facial image data, and the output is analyzed emotion data. This analysis is performed using a deep learning model.

[0744] Step 10:

[0745] Providing feedback

[0746] The server generates additional explanations as needed based on the analyzed emotion data. The input is emotion data, and the output is additional feedback information (audio or text). For example, if the server determines that the user is confused, it generates supplementary explanations and provides them in audio or text.

[0747] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0748] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0749] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0750] [Third embodiment]

[0751] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0752] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0753] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0754] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0755] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0756] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0757] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0758] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0759] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0760] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0761] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0762] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0763] System Overview

[0764] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. The system is composed of the following main components:

[0765] Voice Recognition Module

[0766] Character Recognition Module

[0767] Translation Module

[0768] Video generation and distribution module

[0769] Program processing

[0770] Speech Recognition Module:

[0771] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The captured audio data is converted into a format that is easy for the device to process (e.g., WAV, MP3) and sent to the server. The server passes the received audio data to a speech recognition module, which converts the audio data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[0772] Character Recognition Module:

[0773] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[0774] Translation Module:

[0775] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. The speech recognition "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy", while the character recognition "E = mc²" remains unchanged.

[0776] Video Creation and Distribution Module:

[0777] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles, allowing viewers to understand classes and lectures in their own language. The server distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen.

[0778] Specific examples

[0779] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[0780] This system allows users who speak different languages ​​to understand the same lessons or lectures in real time, making it possible to provide high-quality education and lecture content around the world.

[0781] The processing flow will be explained below.

[0782] Step 1:

[0783] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[0784] Step 2:

[0785] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice data "Today we will talk about the law of conservation of energy" is converted into text data "Today we will talk about the law of conservation of energy."

[0786] Step 3:

[0787] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[0788] Step 4:

[0789] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data: "E = mc²."

[0790] Step 5:

[0791] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, the text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[0792] Step 6:

[0793] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[0794] Step 7:

[0795] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[0796] Example 1

[0797] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0798] Online classes and lectures can be difficult for participants to understand due to language differences. It is particularly important to provide high-quality translations in real time to accurately convey the content to audiences who speak different languages, but this is difficult with current technology.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0800] In this invention, the server includes means for receiving audio data of a class or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, and means for distributing the generated video in real time, thereby enabling viewers who speak different languages ​​to understand the content of the class or lecture in real time.

[0801] "Audio data" refers to electronically configured audio signals obtained from classes, lectures, etc.

[0802] "Text data" is an electronic representation of character information extracted from audio data or video data.

[0803] "Whiteboard or blackboard video data" refers to electronically constructed visual information captured by a camera of what was written during class.

[0804] "Text information" refers to text extracted from video data, and includes mathematical expressions, words, and the like.

[0805] "Real-time" refers to processing or operations that occur at nearly instantaneous speeds.

[0806] "Optical character recognition technology" refers to technology that identifies and reads character information from an image.

[0807] "Subtitle information" refers to translated text data and text that overlays character information on video.

[0808] The "HTTP Live Streaming Protocol" is a communications standard for delivering media content in real time over the Internet.

[0809] "Server" refers to a computer system that processes, translates, and distributes audio and video data.

[0810] "Terminal" refers to an electronic device for capturing and playing audio and video.

[0811] "User" refers to the lecturer who gives the class or lecture, and the participants who watch and listen to it.

[0812] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. Specific embodiments of this system are described in detail below.

[0813] The system consists of three main components: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules are implemented using standard computers and network equipment.

[0814] Speech Recognition Module:

[0815] When a user starts an online class or lecture, the device uses a microphone to capture the user's voice data in real time. The captured voice data is converted by the device into an easy-to-process format such as WAV format and sent to the server. The server then uses a voice recognition module such as Google Cloud Speech-to-Text to convert the voice data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the server converts it into text as "Today I will talk about the law of conservation of energy."

[0816] Character Recognition Module:

[0817] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then uses a character recognition module such as Tesseract OCR to extract text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[0818] Translation Module:

[0819] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[0820] Video Creation and Distribution Module:

[0821] The server uses video generation software such as FFmpeg to generate new video based on the translated text data. The translated text data is overlaid on the generated video as subtitles. This method allows viewers to understand classes and lectures in their own language. The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The viewer's device receives the distributed video and displays it on the user's screen.

[0822] Specific examples

[0823] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to a server. The server uses Google Cloud Speech-to-Text to convert the voice data into text and Tesseract OCR to recognize the mathematical formulas and words written on the blackboard. This data is then translated into English using the Google Translate API, and a video with English subtitles is then generated using FFmpeg and distributed in real time to students overseas. Overseas students can then take the class with English subtitles on their own devices.

[0824] Prompt Sentence Examples

[0825] "Design a program for a system that generates and distributes video with English subtitles in real time based on Japanese audio data."

[0826] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0827] Processing Step Description

[0828] Step 1:

[0829] When a user starts an online class or lecture, the device connects to the system and uses a microphone to capture audio data in real time. The input is the user's voice, and the output is the captured audio data. Specifically, the device launches the audio capture software and collects the lecturer's words as audio data.

[0830] Step 2:

[0831] The device converts the captured audio data to WAV format and sends it to the server using the HTTP protocol. The input is the captured audio data and the output is WAV format audio data. Specifically, the device uses an audio format conversion library to convert the audio data to WAV format and a network library to send the data to the server.

[0832] Step 3:

[0833] The server uses a speech recognition module such as Google Cloud Speech-to-Text to convert the received WAV-formatted audio data into text data. The input is WAV-formatted audio data, and the output is text data. Specifically, the server calls a speech recognition API to convert the audio data into text.

[0834] Step 4:

[0835] While a user is writing on a whiteboard or blackboard, the device's camera captures video data and sends it to the server. The input is the video data of the whiteboard or blackboard, and the output is the captured video data. Specifically, the device launches video capture software, collects video data as frames, converts them into the target format (e.g., H.264), and sends them to the server.

[0836] Step 5:

[0837] The server uses a character recognition module such as Tesseract OCR to extract text information from the received video data. The input is the video data, and the output is the extracted text information. Specifically, the server runs OCR software to analyze the video data and extract the text information as text data.

[0838] Step 6:

[0839] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API to translate it into the desired language. The input is text data, and the output is translated text data. Specifically, the server calls the translation API to convert the text data into the specified language.

[0840] Step 7:

[0841] The server uses video generation software such as FFmpeg to generate a new video with the translated text data overlaid as subtitles. The input is the translated text data and the original video data, and the output is a new video with the subtitle information overlaid. Specifically, the server uses video editing software to overlay the text data onto the video and generate a new video file.

[0842] Step 8:

[0843] The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The input is the subtitled video, and the output is the video delivered to the viewer's device. Specifically, the server uses streaming server software to distribute the video file in live streaming format, and the viewer's device receives and plays the video.

[0844] (Application example 1)

[0845] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0846] In conventional online classes and lectures, it has been difficult for viewers who speak different languages ​​to understand the same content at the same time. Another problem is that real-time translation and subtitle generation cannot be performed with sufficient accuracy. Furthermore, there has been a lack of appropriate systems for delivering video in a format appropriate for the device used by the viewer.

[0847] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0848] In this invention, the server includes a means for receiving audio data of a lecture or class in real time and converting it into text data, a means for extracting video data from a whiteboard or blackboard using optical character recognition technology, a means for translating the text data and the character information into a desired language, a means for generating a video based on the translated text data and the translated character information and distributing it to a user's device in real time, and a means for displaying the generated video on the user's device. This allows viewers who speak different languages ​​to simultaneously understand the same content. Furthermore, viewers can receive the appropriately distributed video in real time using devices such as smartphones or head-mounted displays.

[0849] A "class or lecture" is a teaching or informational session held at an educational institution, business, etc.

[0850] "Audio data" is digital information of sound captured by a sound capturing device such as a microphone.

[0851] "Text data" refers to character information converted from audio data or video data.

[0852] "Video data" refers to digital information of video captured by a video camera or other video capture device.

[0853] Optical character recognition (OCR) is a technology for extracting characters from images and converting handwritten or printed characters into digital text.

[0854] "Translation means" refers to the process and techniques for converting text data expressed in one language into another language.

[0855] "Means for generating images" refers to the processes and technologies for creating digital images based on translated text data or character information.

[0856] "User device" refers to an electronic device used by a user to receive and display information, such as a smartphone or head-mounted display.

[0857] "Real-time delivery means" refers to the process and technology for instantly transmitting the generated video to the user's terminal.

[0858] This invention is a system that receives audio and video data of classes and lectures in real time, translates it into multiple languages, and distributes it to viewers. The system consists of the following main modules: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules work in combination to enable users who speak different languages ​​to simultaneously understand the same content.

[0859] Voice Recognition Module

[0860] Hardware: Smartphone microphone, head-mounted display microphone

[0861] Software: Speech recognition API (e.g., Google Cloud Speech-to-Text API)

[0862] Processing details: Audio data from classes and lectures is captured in real time and converted into text using a speech recognition API. For example, if a lecturer says, "Today, I'll talk about the law of conservation of energy," it will be converted into text as, "Today, I'll talk about the law of conservation of energy."

[0863] Character Recognition Module

[0864] Hardware: Smartphone camera, head-mounted display camera

[0865] Software: Optical Character Recognition technology (OCR, e.g. Google Cloud Vision API)

[0866] What it does: Captures text written on a whiteboard or blackboard and uses OCR to extract text information, such as "E = mc²," and converts it into digital text.

[0867] Translation Module

[0868] Software: Translation API (e.g., Google Cloud Translation API)

[0869] Processing content: Translate text data acquired through speech recognition or character recognition into the desired language. For example, "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[0870] Video generation and distribution module

[0871] Software: Video generation software (e.g., FFmpeg), real-time streaming software (e.g., WebRTC)

[0872] Processing details: Generates video based on translated text data and overlays subtitles. The generated video is delivered to the user's device in real time and displayed on the user's smartphone or head-mounted display.

[0873] Specific examples

[0874] When a user takes an online class in Japan, their smartphone or head-mounted display captures the instructor's voice and the contents of the blackboard. The server converts the voice into text using a speech recognition module and recognizes the contents of the blackboard using a character recognition module. This information is then translated into English using a translation module, and a video with English subtitles is generated and distributed in real time using a video generation and distribution module. Students overseas can take classes with English subtitles on their smartphones or head-mounted displays.

[0875] Prompt Sentence Examples

[0876] Please create a program that recognizes the audio and blackboard text of a Japanese online lecture in real time, translates it into English, and distributes it as a video with subtitles. Specific requirements are as follows:

[0877] 1. Convert audio to text using the Google Cloud Speech-to-Text API.

[0878] 2. The characters on the blackboard are recognized using Google Cloud Vision API (OCR function).

[0879] 3. Translation uses Google Cloud Translation API.

[0880] 4. FFmpeg is used to generate video and WebRTC is used for real-time streaming.

[0881] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0882] Step 1:

[0883] The device uses a microphone to capture audio data from classes and lectures in real time.

[0884] Input: The instructor's voice

[0885] Data processing: Converting audio data into digital format (e.g. WAV format)

[0886] Output: Digital audio data

[0887] Step 2:

[0888] The terminal transmits the captured audio data to a server over the Internet.

[0889] Input: Digital audio data

[0890] Data Computing: Data Transfer via the Internet

[0891] Output: Audio data received on the server

[0892] Step 3:

[0893] The server calls a speech recognition API to convert the speech data into text data.

[0894] Input: Audio data received on the server

[0895] Data processing: Convert to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text API).

[0896] Output: Text data generated by speech recognition

[0897] Step 4:

[0898] The device uses a camera to capture the contents of the whiteboard or blackboard.

[0899] Input: What is written on a whiteboard or blackboard

[0900] Data processing: Converting video data into digital format (e.g., JPEG or MP4 format)

[0901] Output: Digital video data

[0902] Step 5:

[0903] The terminal transmits the captured video data to a server via the Internet.

[0904] Input: Digital video data

[0905] Data Computing: Data Transfer via the Internet

[0906] Output: Video data received on the server

[0907] Step 6:

[0908] The server extracts text information from the video data using optical character recognition (OCR) technology.

[0909] Input: Video data received on the server

[0910] Data processing: Convert to text data using OCR technology (e.g., Google Cloud Vision API)

[0911] Output: Character information extracted by OCR

[0912] Step 7:

[0913] The server translates the text data generated by the speech recognition and character recognition into a desired language using a translation API.

[0914] Input: Speech recognition text data, character recognition text data

[0915] Data processing: Translate into the desired language using a translation API (e.g., Google Cloud Translation API)

[0916] Output: Translated text data

[0917] Step 8:

[0918] The server generates a video using the translated text data and overlays the subtitles onto the video.

[0919] Input: Translated text data

[0920] Data processing: Generate subtitled video using video generation software (e.g. FFmpeg)

[0921] Output: Video data with generated subtitles

[0922] Step 9:

[0923] The server distributes the generated subtitled video to the user's terminal in real time.

[0924] Input: Generated video data with subtitles

[0925] Data calculation: Transmitted to the user's device using real-time delivery technology (e.g., WebRTC)

[0926] Output: Subtitled video played on the user's device

[0927] Step 10:

[0928] The user's terminal displays the subtitled video delivered in real time.

[0929] Input: Video data with subtitles delivered in real time

[0930] Data processing: decoding and displaying video data

[0931] Output: Video with translated subtitles displayed on the user's device screen

[0932] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0933] System Overview

[0934] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[0935] Voice Recognition Module

[0936] Character Recognition Module

[0937] Translation Module

[0938] Video generation and distribution module

[0939] Emotion Engine

[0940] Program processing

[0941] Speech Recognition Module:

[0942] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[0943] Character Recognition Module:

[0944] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[0945] Translation Module:

[0946] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[0947] Video Creation and Distribution Module:

[0948] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle with the text "Today we will discuss the law of energy conservation" could be added. The server then distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen. Viewers can then follow lessons and lectures in their own language.

[0949] Emotion Engine:

[0950] A camera installed on the user's device captures the user's facial expression. The captured video data is sent to the server, which then passes it to the emotion engine. The emotion engine analyzes the user's facial expression and recognizes the emotion. The recognized emotion is fed back to the translation module and reflected in the translation method and image generation. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[0951] Specific examples

[0952] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[0953] Furthermore, the system captures the user's facial expressions in real time and uses an emotion engine to analyze their emotions. For example, if the system determines that the user is confused, it will provide additional explanations. In this way, high-quality educational and lecture content can be delivered worldwide, regardless of language barriers, to deepen user understanding.

[0954] The processing flow will be explained below.

[0955] Step 1:

[0956] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[0957] Step 2:

[0958] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice "Today we will talk about the law of conservation of energy" is analyzed and converted into text data "Today we will talk about the law of conservation of energy."

[0959] Step 3:

[0960] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard in real time. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[0961] Step 4:

[0962] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data "E = mc²."

[0963] Step 5:

[0964] A camera installed on the user's device captures the user's facial expression. The captured facial data is sent to the server in real time. The server passes the received facial data to the emotion engine, which then analyzes the user's emotions. For example, if the user has a confused expression, the emotion engine will recognize this.

[0965] Step 6:

[0966] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy." "E = mc²" is left as is.

[0967] Step 7:

[0968] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[0969] Step 8:

[0970] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[0971] Step 9:

[0972] The emotion engine analyzes the user's emotion data and feeds it back to the translation module. For example, if the user looks confused, the translation module changes to provide a translation in simpler terms. Based on this emotion feedback, the server regenerates the video and delivers it with additional information to deepen the user's understanding.

[0973] In this way, the system translates audio and video content in real time, and even analyzes and provides feedback on the user's emotions, thereby enabling high-quality education and information provision.

[0974] Example 2

[0975] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0976] In online classes and lectures, real-time translation of audio and video content is important for overcoming language barriers. However, conventional systems have had difficulty not only translating audio and video data, but also appropriately reflecting the user's level of understanding and emotions. Furthermore, there was no way to analyze the user's emotions in real time and adjust the system's behavior based on that, making it difficult to provide efficient and effective feedback.

[0977] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the voice data into text data, means for receiving video data, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for capturing a user's facial expression and analyzing the emotion, and means for adjusting the system operation based on the analyzed emotion. This makes it possible not only to translate voice and video data in real time, but also to analyze the user's emotion and provide appropriate feedback.

[0978] "Audio data" is data that represents an audio signal in digital form.

[0979] "Text data" is character string data extracted from audio data or video data.

[0980] "Video data" refers to video data captured by a device such as a camera.

[0981] "Text information" refers to information such as characters and symbols extracted from video data.

[0982] "Translation" is the process of converting text data expressed in one language into another language.

[0983] "Distribution" means transmitting the generated video to viewers in real time via a network.

[0984] "User's facial expression" refers to the facial expression of the user captured by the camera.

[0985] "Analyzing emotions" refers to a process of identifying the emotions felt by the user based on the user's facial expression data.

[0986] "Adjusting the system's behavior" means changing the system's functions and behavior based on the analyzed emotional information.

[0987] System Overview

[0988] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[0989] Voice Recognition Module

[0990] Character Recognition Module

[0991] Translation Module

[0992] Video generation and distribution module

[0993] Emotion Engine

[0994] Voice Recognition Module

[0995] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data is converted into a format such as WAV or MP3 and sent to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," it will be converted into text data.

[0996] Character Recognition Module

[0997] When a user is teaching a lesson while writing on a whiteboard or blackboard, the device's camera captures the video. The captured video data is also sent to the server, and the server passes the received video data to a character recognition module (OCR). The character recognition module extracts character information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[0998] Translation Module

[0999] The server passes the text data extracted from the speech recognition module and character recognition module to the translation module, which translates the text data into the specified language. For example, "Today we will talk about the law of conservation of energy" is translated into English as "Today we will discuss the law of conservation of energy." The mathematical formula "E = mc²" is kept as is.

[1000] Video generation and distribution module

[1001] The server generates a new video based on the translated text data. This new video is overlaid with subtitles of the translated text. The generated video is then distributed to viewers in real time. Viewers receive this video on their own devices and display it on their screens.

[1002] Emotion Engine

[1003] The user's device uses a built-in camera to capture the user's facial expressions. The captured video data is sent to a server, which then passes it to an emotion engine to analyze the user's facial expressions. The emotion engine recognizes the user's emotions and feeds that information back to the system. Based on this feedback, the system can reflect the information in translations and video generation, if necessary. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[1004] Specific examples

[1005] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to the server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by the translation module, and then the video generation and distribution module generates a video with English subtitles and distributes it in real time to overseas students. The overseas students can then take the class with English subtitles on their own devices. Furthermore, the system captures the user's facial expressions in real time and analyzes their emotions with an emotion engine. For example, if the system determines that the user is confused, it provides additional supplementary explanations. In this way, high-quality education and lecture content can be provided worldwide, transcending language barriers and deepening users' understanding.

[1006] Example prompts for generative AI models

[1007] Example prompt 1:

[1008] "Please translate the Japanese audio data into English and distribute it with English subtitles."

[1009] Example prompt 2:

[1010] "Please recognize the mathematical formulas written on the blackboard and add them directly to the English subtitles."

[1011] Example prompt 3:

[1012] "Analyze the user's facial expression and display additional explanatory text if confused."

[1013] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1014] Step 1:

[1015] Terminal

[1016] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data (input) is converted into a format such as WAV or MP3 (data processing) and then sent to the server (output).

[1017] Specific behavior:

[1018] When a user starts speaking, "Today I'm going to talk about the law of conservation of energy," the audio is captured by the device's microphone. The captured audio data is converted into WAV format and sent from the device to the server.

[1019] Step 2:

[1020] server

[1021] The server passes the voice data (input) received from the terminal to the voice recognition module. This voice recognition module converts the voice data into character string data (text data) (data conversion). The character string data (output) is stored within the server.

[1022] Specific behavior:

[1023] When the server inputs the received voice data into the voice recognition module, the voice data "Today we will talk about the law of conservation of energy" is converted into text "Today we will talk about the law of conservation of energy."

[1024] Step 3:

[1025] Terminal

[1026] When a user is teaching a class while writing on a whiteboard or blackboard, the device camera captures this video. The captured video data (input) is sent to the server (output).

[1027] Specific behavior:

[1028] When a user writes "E = mc²" on the blackboard, the device's camera captures the image and the image data is sent to the server.

[1029] Step 4:

[1030] server

[1031] The server passes the received video data (input) to a character recognition module (OCR). This character recognition module extracts character information from the video data (data extraction). The extracted character information (output) is stored within the server.

[1032] Specific behavior:

[1033] When the video data received by the server is input into the character recognition module, the character information "E = mc²" is extracted and saved as text data.

[1034] Step 5:

[1035] server

[1036] The server passes the text data (input) obtained from the speech recognition module and character recognition module to the translation module. The translation module translates this data into the desired language (data translation). The translated text data (output) is stored within the server.

[1037] Specific behavior:

[1038] The text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy", and the text data "E = mc²" is left as is.

[1039] Step 6:

[1040] server

[1041] The server uses a video generation and distribution module to generate new video based on the translated text data (input). This module overlays the translated text data as subtitles (data generation). The generated video (output) is distributed to viewers in real time.

[1042] Specific behavior:

[1043] The server generates a video with an overlay of the subtitle "Today we will discuss the law of conservation of energy" and distributes it to viewers in real time.

[1044] Step 7:

[1045] Terminal

[1046] The user's device captures the user's facial expressions using a built-in camera, and the captured video data (input) is sent to the server (output).

[1047] Specific behavior:

[1048] If the user makes a confused expression, the expression is captured by the device's camera and the video data is sent to the server.

[1049] Step 8:

[1050] server

[1051] The server passes the received video data (input) to the emotion engine, which analyzes the user's facial expressions and recognizes their emotions (data analysis). The analyzed emotions (output) are used to adjust the system's behavior.

[1052] Specific behavior:

[1053] When the server inputs the captured user's facial expressions into an emotion engine, the emotion of confusion is recognized, and the system responds by generating and delivering additional explanatory text.

[1054] (Application example 2)

[1055] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1056] Conventional online classes and lecture systems have had problems with real-time multilingual translation and providing feedback based on the user's level of understanding. Specifically, it is difficult to ensure that students and audience members who speak different languages ​​can instantly understand the content of the lecture. They also lack the functionality to supplement the content of the lecture based on the user's facial expressions and emotions. This has led to issues such as a decline in the quality of the classes and lectures and a decrease in the user's learning effectiveness.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1058] In this invention, the server includes means for receiving audio data of a lesson or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for receiving user facial expression data and analyzing the emotions, and means for providing additional explanations based on the analyzed emotion data. This makes it possible for users who speak different languages ​​to understand the content of the lesson in real time, and furthermore, by providing feedback based on the user's emotions, it is possible to improve the quality of the lesson or lecture and increase the user's learning effectiveness.

[1059] "Means for receiving audio data" refers to a device or function for capturing the audio of a class or lecture, converting it into digital data, and transmitting it to a server.

[1060] The "means for converting voice data into text data" refers to a voice recognition technology for analyzing received voice data and extracting its contents as text information.

[1061] "Means for receiving video data" refers to a device or function that captures what is written on a whiteboard or blackboard using a camera or other device and transmits it to a server as digital data.

[1062] The "means for extracting character information from video data" is a means for analyzing captured video data, recognizing characters therein, and converting them into text data.

[1063] "Means for translating text data and character information into a desired language" refers to a translation technology for converting text data obtained by speech recognition or character recognition into a language desired by the user.

[1064] "Means for generating video based on translated text data and translated character information" refers to video generation technology for creating new video based on translated text data and adding subtitles, etc. to that video.

[1065] The "means for distributing the generated video" refers to a network and software for distributing the generated video data to users in real time.

[1066] The "means for receiving the user's facial expression data and analyzing the emotions" is a technology for capturing the user's facial image in real time, analyzing the emotions, and acquiring them as data.

[1067] The "means for providing additional explanations based on analyzed emotion data" is an automatic response system that provides supplementary explanations of the content of a class or lecture based on the analyzed emotion of the user.

[1068] This invention relates to a system that performs real-time translation and sentiment analysis for online classes and lectures. This system consists of the following steps and components:

[1069] First, the server has a means for receiving the audio data of the class or lecture. This includes devices and functions that capture audio using a microphone installed on the terminal, convert the audio data into a digital format, and send it to the server. The server then converts the received audio data into text data using speech recognition technology.

[1070] Next, the server is equipped with a means for receiving video data. This includes a device or function that captures what is written on the whiteboard or blackboard with a camera, converts the video data into a digital format, and transmits it to the server. The server then extracts text information from the received video data using optical character recognition (OCR) technology.

[1071] Furthermore, the server translates the text data obtained by speech recognition and character recognition into the desired language using multilingual translation technology, such as the Google Translate API.

[1072] The server generates a new video based on the translated text data and the translated character information. This video generation includes technology to overlay the text data as subtitles. The generated video is delivered to the user's device in real time.

[1073] Furthermore, the server is equipped with a means for receiving the user's facial expression data and analyzing the emotions. The device uses a camera to capture the user's face and transmits the facial expression data to the server. The server then uses an emotion recognition algorithm to analyze the user's emotions in real time. For example, facial recognition technology and deep learning models can be used for emotion recognition.

[1074] Based on the analyzed emotional data, the server provides appropriate feedback. For example, if it determines that the user is confused, it provides additional explanations via voice or text. This feedback deepens the user's understanding and improves the quality of lessons or lectures.

[1075] A specific example of this application is a language learning platform. Using this system, users can understand lectures from instructors in multiple languages ​​in real time. Furthermore, if users have questions, additional explanations are automatically provided, improving learning effectiveness. This system is implemented using technologies such as Google Translate API, OpenCV, SpeechRecognition, and Pyttsx3.

[1076] Example prompt sentence:

[1077] "Please translate what is being said in Japanese into English and display it as subtitles on the screen. Additionally, please analyze the user's facial expressions and provide additional explanations if they are confused."

[1078] In this way, we provide a system that can perform translation and emotion analysis in real time to support user understanding.

[1079] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1080] Step 1:

[1081] Capture audio data

[1082] The device captures the audio data of a class or lecture using a microphone. The input is the teacher's or lecturer's audio data, and the output is audio data in a digital format (e.g., WAV, MP3). This digital data is sent to a server.

[1083] Step 2:

[1084] Speech-to-text conversion

[1085] The server passes the received audio data to a speech recognition module. The input is digital audio data, and the output is the corresponding text data. This conversion is done using a speech recognition library such as SpeechRecognition.

[1086] Step 3:

[1087] Video data capture

[1088] The device captures video data of the whiteboard or blackboard with a camera. The input is what the teacher or lecturer has written, and the output is digital video data. This digital data is sent to the server.

[1089] Step 4:

[1090] Extracting text information from video

[1091] The server passes the received video data to a character recognition module. The input is digital video data, and the output is extracted text information. This conversion is performed using optical character recognition (OCR) technology.

[1092] Step 5:

[1093] Text data translation

[1094] The server passes the text data from the speech recognition module and the character information from the character recognition module to the translation module. The input is the text data and character information, and the output is the text data translated into the desired language. This translation is performed using multilingual translation technologies such as DeepL or Google Translate API.

[1095] Step 6:

[1096] Video generation

[1097] The server generates new videos based on the translated text data. The input is the translated text data and character information, and the output is video data with subtitles added. Video generation technology is used for this generation.

[1098] Step 7:

[1099] Video distribution

[1100] The server distributes the generated video data to viewers in real time. The input is the generated video data, and the output is the video displayed on the user's device.

[1101] Step 8:

[1102] Capturing facial expression data

[1103] The device captures the user's facial expression with a camera. The input is the user's facial image data, and the output is the digital data. This digital data is sent to the server.

[1104] Step 9:

[1105] Emotion Analysis

[1106] The server passes the received facial expression data to the emotion engine. The input is digital facial image data, and the output is analyzed emotion data. This analysis is performed using a deep learning model.

[1107] Step 10:

[1108] Providing feedback

[1109] The server generates additional explanations as needed based on the analyzed emotion data. The input is emotion data, and the output is additional feedback information (audio or text). For example, if the server determines that the user is confused, it generates supplementary explanations and provides them in audio or text.

[1110] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1111] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1112] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1113] [Fourth embodiment]

[1114] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1115] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1116] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1117] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1118] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1119] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1120] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1121] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1122] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1123] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1124] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1125] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1126] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1127] System Overview

[1128] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. The system is composed of the following main components:

[1129] Voice Recognition Module

[1130] Character Recognition Module

[1131] Translation Module

[1132] Video generation and distribution module

[1133] Program processing

[1134] Speech Recognition Module:

[1135] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The captured audio data is converted into a format that is easy for the device to process (e.g., WAV, MP3) and sent to the server. The server passes the received audio data to a speech recognition module, which converts the audio data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[1136] Character Recognition Module:

[1137] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[1138] Translation Module:

[1139] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. The speech recognition "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy", while the character recognition "E = mc²" remains unchanged.

[1140] Video Creation and Distribution Module:

[1141] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles, allowing viewers to understand classes and lectures in their own language. The server distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen.

[1142] Specific examples

[1143] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[1144] This system allows users who speak different languages ​​to understand the same lessons or lectures in real time, making it possible to provide high-quality education and lecture content around the world.

[1145] The processing flow will be explained below.

[1146] Step 1:

[1147] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[1148] Step 2:

[1149] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice data "Today we will talk about the law of conservation of energy" is converted into text data "Today we will talk about the law of conservation of energy."

[1150] Step 3:

[1151] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[1152] Step 4:

[1153] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data: "E = mc²."

[1154] Step 5:

[1155] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, the text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[1156] Step 6:

[1157] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[1158] Step 7:

[1159] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[1160] Example 1

[1161] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1162] Online classes and lectures can be difficult for participants to understand due to language differences. It is particularly important to provide high-quality translations in real time to accurately convey the content to audiences who speak different languages, but this is difficult with current technology.

[1163] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1164] In this invention, the server includes means for receiving audio data of a class or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, and means for distributing the generated video in real time, thereby enabling viewers who speak different languages ​​to understand the content of the class or lecture in real time.

[1165] "Audio data" refers to electronically configured audio signals obtained from classes, lectures, etc.

[1166] "Text data" is an electronic representation of character information extracted from audio data or video data.

[1167] "Whiteboard or blackboard video data" refers to electronically constructed visual information captured by a camera of what was written during class.

[1168] "Text information" refers to text extracted from video data, and includes mathematical expressions, words, and the like.

[1169] "Real-time" refers to processing or operations that occur at nearly instantaneous speeds.

[1170] "Optical character recognition technology" refers to technology that identifies and reads character information from an image.

[1171] "Subtitle information" refers to translated text data and text that overlays character information on video.

[1172] The "HTTP Live Streaming Protocol" is a communications standard for delivering media content in real time over the Internet.

[1173] "Server" refers to a computer system that processes, translates, and distributes audio and video data.

[1174] "Terminal" refers to an electronic device for capturing and playing audio and video.

[1175] "User" refers to the lecturer who gives the class or lecture, and the participants who watch and listen to it.

[1176] This invention is a system that translates audio and video content in real time during online classes and lectures, enabling education and information provision that transcends national and language barriers. Specific embodiments of this system are described in detail below.

[1177] The system consists of three main components: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules are implemented using standard computers and network equipment.

[1178] Speech Recognition Module:

[1179] When a user starts an online class or lecture, the device uses a microphone to capture the user's voice data in real time. The captured voice data is converted by the device into an easy-to-process format such as WAV format and sent to the server. The server then uses a voice recognition module such as Google Cloud Speech-to-Text to convert the voice data into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the server converts it into text as "Today I will talk about the law of conservation of energy."

[1180] Character Recognition Module:

[1181] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then uses a character recognition module such as Tesseract OCR to extract text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[1182] Translation Module:

[1183] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[1184] Video Creation and Distribution Module:

[1185] The server uses video generation software such as FFmpeg to generate new video based on the translated text data. The translated text data is overlaid on the generated video as subtitles. This method allows viewers to understand classes and lectures in their own language. The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The viewer's device receives the distributed video and displays it on the user's screen.

[1186] Specific examples

[1187] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to a server. The server uses Google Cloud Speech-to-Text to convert the voice data into text and Tesseract OCR to recognize the mathematical formulas and words written on the blackboard. This data is then translated into English using the Google Translate API, and a video with English subtitles is then generated using FFmpeg and distributed in real time to students overseas. Overseas students can then take the class with English subtitles on their own devices.

[1188] Prompt Sentence Examples

[1189] "Design a program for a system that generates and distributes video with English subtitles in real time based on Japanese audio data."

[1190] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1191] Processing Step Description

[1192] Step 1:

[1193] When a user starts an online class or lecture, the device connects to the system and uses a microphone to capture audio data in real time. The input is the user's voice, and the output is the captured audio data. Specifically, the device launches the audio capture software and collects the lecturer's words as audio data.

[1194] Step 2:

[1195] The device converts the captured audio data to WAV format and sends it to the server using the HTTP protocol. The input is the captured audio data and the output is WAV format audio data. Specifically, the device uses an audio format conversion library to convert the audio data to WAV format and a network library to send the data to the server.

[1196] Step 3:

[1197] The server uses a speech recognition module such as Google Cloud Speech-to-Text to convert the received WAV-formatted audio data into text data. The input is WAV-formatted audio data, and the output is text data. Specifically, the server calls a speech recognition API to convert the audio data into text.

[1198] Step 4:

[1199] While a user is writing on a whiteboard or blackboard, the device's camera captures video data and sends it to the server. The input is the video data of the whiteboard or blackboard, and the output is the captured video data. Specifically, the device launches video capture software, collects video data as frames, converts them into the target format (e.g., H.264), and sends them to the server.

[1200] Step 5:

[1201] The server uses a character recognition module such as Tesseract OCR to extract text information from the received video data. The input is the video data, and the output is the extracted text information. Specifically, the server runs OCR software to analyze the video data and extract the text information as text data.

[1202] Step 6:

[1203] The server passes the text data from the speech recognition module and the text data from the character recognition module to a translation module such as Google Translate API to translate it into the desired language. The input is text data, and the output is translated text data. Specifically, the server calls the translation API to convert the text data into the specified language.

[1204] Step 7:

[1205] The server uses video generation software such as FFmpeg to generate a new video with the translated text data overlaid as subtitles. The input is the translated text data and the original video data, and the output is a new video with the subtitle information overlaid. Specifically, the server uses video editing software to overlay the text data onto the video and generate a new video file.

[1206] Step 8:

[1207] The server then distributes the generated video to viewers in real time using protocols such as HTTP Live Streaming (HLS). The input is the subtitled video, and the output is the video delivered to the viewer's device. Specifically, the server uses streaming server software to distribute the video file in live streaming format, and the viewer's device receives and plays the video.

[1208] (Application example 1)

[1209] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1210] In conventional online classes and lectures, it has been difficult for viewers who speak different languages ​​to understand the same content at the same time. Another problem is that real-time translation and subtitle generation cannot be performed with sufficient accuracy. Furthermore, there has been a lack of appropriate systems for delivering video in a format appropriate for the device used by the viewer.

[1211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1212] In this invention, the server includes a means for receiving audio data of a lecture or class in real time and converting it into text data, a means for extracting video data from a whiteboard or blackboard using optical character recognition technology, a means for translating the text data and the character information into a desired language, a means for generating a video based on the translated text data and the translated character information and distributing it to a user's device in real time, and a means for displaying the generated video on the user's device. This allows viewers who speak different languages ​​to simultaneously understand the same content. Furthermore, viewers can receive the appropriately distributed video in real time using devices such as smartphones or head-mounted displays.

[1213] A "class or lecture" is a teaching or informational session held at an educational institution, business, etc.

[1214] "Audio data" is digital information of sound captured by a sound capturing device such as a microphone.

[1215] "Text data" refers to character information converted from audio data or video data.

[1216] "Video data" refers to digital information of video captured by a video camera or other video capture device.

[1217] Optical character recognition (OCR) is a technology for extracting characters from images and converting handwritten or printed characters into digital text.

[1218] "Translation means" refers to the process and techniques for converting text data expressed in one language into another language.

[1219] "Means for generating images" refers to the processes and technologies for creating digital images based on translated text data or character information.

[1220] "User device" refers to an electronic device used by a user to receive and display information, such as a smartphone or head-mounted display.

[1221] "Real-time delivery means" refers to the process and technology for instantly transmitting the generated video to the user's terminal.

[1222] This invention is a system that receives audio and video data of classes and lectures in real time, translates it into multiple languages, and distributes it to viewers. The system consists of the following main modules: a speech recognition module, a character recognition module, a translation module, and a video generation and distribution module. These modules work in combination to enable users who speak different languages ​​to simultaneously understand the same content.

[1223] Voice Recognition Module

[1224] Hardware: Smartphone microphone, head-mounted display microphone

[1225] Software: Speech recognition API (e.g., Google Cloud Speech-to-Text API)

[1226] Processing details: Audio data from classes and lectures is captured in real time and converted into text using a speech recognition API. For example, if a lecturer says, "Today, I'll talk about the law of conservation of energy," it will be converted into text as, "Today, I'll talk about the law of conservation of energy."

[1227] Character Recognition Module

[1228] Hardware: Smartphone camera, head-mounted display camera

[1229] Software: Optical Character Recognition technology (OCR, e.g. Google Cloud Vision API)

[1230] What it does: Captures text written on a whiteboard or blackboard and uses OCR to extract text information, such as "E = mc²," and converts it into digital text.

[1231] Translation Module

[1232] Software: Translation API (e.g., Google Cloud Translation API)

[1233] Processing content: Translate text data acquired through speech recognition or character recognition into the desired language. For example, "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy."

[1234] Video generation and distribution module

[1235] Software: Video generation software (e.g., FFmpeg), real-time streaming software (e.g., WebRTC)

[1236] Processing details: Generates video based on translated text data and overlays subtitles. The generated video is delivered to the user's device in real time and displayed on the user's smartphone or head-mounted display.

[1237] Specific examples

[1238] When a user takes an online class in Japan, their smartphone or head-mounted display captures the instructor's voice and the contents of the blackboard. The server converts the voice into text using a speech recognition module and recognizes the contents of the blackboard using a character recognition module. This information is then translated into English using a translation module, and a video with English subtitles is generated and distributed in real time using a video generation and distribution module. Students overseas can take classes with English subtitles on their smartphones or head-mounted displays.

[1239] Prompt Sentence Examples

[1240] Please create a program that recognizes the audio and blackboard text of a Japanese online lecture in real time, translates it into English, and distributes it as a video with subtitles. Specific requirements are as follows:

[1241] 1. Convert audio to text using the Google Cloud Speech-to-Text API.

[1242] 2. The characters on the blackboard are recognized using Google Cloud Vision API (OCR function).

[1243] 3. Translation uses Google Cloud Translation API.

[1244] 4. FFmpeg is used to generate video and WebRTC is used for real-time streaming.

[1245] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1246] Step 1:

[1247] The device uses a microphone to capture audio data from classes and lectures in real time.

[1248] Input: The instructor's voice

[1249] Data processing: Converting audio data into digital format (e.g. WAV format)

[1250] Output: Digital audio data

[1251] Step 2:

[1252] The terminal transmits the captured audio data to a server over the Internet.

[1253] Input: Digital audio data

[1254] Data Computing: Data Transfer via the Internet

[1255] Output: Audio data received on the server

[1256] Step 3:

[1257] The server calls a speech recognition API to convert the speech data into text data.

[1258] Input: Audio data received on the server

[1259] Data processing: Convert to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text API).

[1260] Output: Text data generated by speech recognition

[1261] Step 4:

[1262] The device uses a camera to capture the contents of the whiteboard or blackboard.

[1263] Input: What is written on a whiteboard or blackboard

[1264] Data processing: Converting video data into digital format (e.g., JPEG or MP4 format)

[1265] Output: Digital video data

[1266] Step 5:

[1267] The terminal transmits the captured video data to a server via the Internet.

[1268] Input: Digital video data

[1269] Data Computing: Data Transfer via the Internet

[1270] Output: Video data received on the server

[1271] Step 6:

[1272] The server extracts text information from the video data using optical character recognition (OCR) technology.

[1273] Input: Video data received on the server

[1274] Data processing: Convert to text data using OCR technology (e.g., Google Cloud Vision API)

[1275] Output: Character information extracted by OCR

[1276] Step 7:

[1277] The server translates the text data generated by the speech recognition and character recognition into a desired language using a translation API.

[1278] Input: Speech recognition text data, character recognition text data

[1279] Data processing: Translate into the desired language using a translation API (e.g., Google Cloud Translation API)

[1280] Output: Translated text data

[1281] Step 8:

[1282] The server generates a video using the translated text data and overlays the subtitles onto the video.

[1283] Input: Translated text data

[1284] Data processing: Generate subtitled video using video generation software (e.g. FFmpeg)

[1285] Output: Video data with generated subtitles

[1286] Step 9:

[1287] The server distributes the generated subtitled video to the user's terminal in real time.

[1288] Input: Generated video data with subtitles

[1289] Data calculation: Transmitted to the user's device using real-time delivery technology (e.g., WebRTC)

[1290] Output: Subtitled video played on the user's device

[1291] Step 10:

[1292] The user's terminal displays the subtitled video delivered in real time.

[1293] Input: Video data with subtitles delivered in real time

[1294] Data processing: decoding and displaying video data

[1295] Output: Video with translated subtitles displayed on the user's device screen

[1296] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1297] System Overview

[1298] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[1299] Voice Recognition Module

[1300] Character Recognition Module

[1301] Translation Module

[1302] Video generation and distribution module

[1303] Emotion Engine

[1304] Program processing

[1305] Speech Recognition Module:

[1306] When a user starts an online class or lecture, the device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," the module converts it into text: "Today I will talk about the law of conservation of energy."

[1307] Character Recognition Module:

[1308] As a user writes on a whiteboard or blackboard during a lesson, the device's camera captures this and sends the video data to the server. The server then passes the received video data to a character recognition module (OCR) and extracts text information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data: "E = mc²."

[1309] Translation Module:

[1310] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" from the speech recognition module is translated to "Today we will discuss the law of conservation of energy," while "E = mc²" from the character recognition module remains unchanged.

[1311] Video Creation and Distribution Module:

[1312] The server generates new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle with the text "Today we will discuss the law of energy conservation" could be added. The server then distributes the generated video to viewers in real time, and the device receives the distributed video and displays it on the user's screen. Viewers can then follow lessons and lectures in their own language.

[1313] Emotion Engine:

[1314] A camera installed on the user's device captures the user's facial expression. The captured video data is sent to the server, which then passes it to the emotion engine. The emotion engine analyzes the user's facial expression and recognizes the emotion. The recognized emotion is fed back to the translation module and reflected in the translation method and image generation. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[1315] Specific examples

[1316] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard, and sends them to a server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by a translation module, and then a video generation and distribution module generates a video with English subtitles and distributes it in real time to students overseas. Students overseas can then take the class with English subtitles on their own devices.

[1317] Furthermore, the system captures the user's facial expressions in real time and uses an emotion engine to analyze their emotions. For example, if the system determines that the user is confused, it will provide additional explanations. In this way, high-quality educational and lecture content can be delivered worldwide, regardless of language barriers, to deepen user understanding.

[1318] The processing flow will be explained below.

[1319] Step 1:

[1320] A user starts an online class or lecture. The device uses a microphone to capture audio data in real time. The device converts the captured audio data into an easy-to-process format (e.g., WAV, MP3) and sends it to the server.

[1321] Step 2:

[1322] The server passes the received voice data to the voice recognition module. The voice recognition module analyzes the voice data and converts it into text data. For example, the voice "Today we will talk about the law of conservation of energy" is analyzed and converted into text data "Today we will talk about the law of conservation of energy."

[1323] Step 3:

[1324] The user conducts the lesson by writing content on a whiteboard or blackboard. The device uses a camera to capture images of the whiteboard or blackboard in real time. The device converts the captured image data into an easy-to-process format (e.g., JPEG, PNG) and sends it to the server.

[1325] Step 4:

[1326] The server passes the received video data to a character recognition module (OCR). The character recognition module analyzes the video data and extracts character information. For example, if the string "E = mc²" is written on a blackboard, it will be extracted as text data "E = mc²."

[1327] Step 5:

[1328] A camera installed on the user's device captures the user's facial expression. The captured facial data is sent to the server in real time. The server passes the received facial data to the emotion engine, which then analyzes the user's emotions. For example, if the user has a confused expression, the emotion engine will recognize this.

[1329] Step 6:

[1330] The server passes the text data from the speech recognition module and the text data from the character recognition module to the translation module. The translation module translates the text data into the desired language in real time. For example, "Today we will talk about the law of conservation of energy" is translated to "Today we will discuss the law of conservation of energy." "E = mc²" is left as is.

[1331] Step 7:

[1332] The server generates a new video based on the translated audio and text data. The translated text data is overlaid on the generated video as subtitles. For example, a subtitle saying "Today we will discuss the law of energy conservation" could be added.

[1333] Step 8:

[1334] The server distributes the generated video to viewers in real time. The device receives the distributed video and displays it on the user's screen. Viewers can follow lessons and lectures in their own language.

[1335] Step 9:

[1336] The emotion engine analyzes the user's emotion data and feeds it back to the translation module. For example, if the user looks confused, the translation module changes to provide a translation in simpler terms. Based on this emotion feedback, the server regenerates the video and delivers it with additional information to deepen the user's understanding.

[1337] In this way, the system translates audio and video content in real time, and even analyzes and provides feedback on the user's emotions, thereby enabling high-quality education and information provision.

[1338] Example 2

[1339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1340] In online classes and lectures, real-time translation of audio and video content is important for overcoming language barriers. However, conventional systems have had difficulty not only translating audio and video data, but also appropriately reflecting the user's level of understanding and emotions. Furthermore, there was no way to analyze the user's emotions in real time and adjust the system's behavior based on that, making it difficult to provide efficient and effective feedback.

[1341] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving voice data, means for converting the voice data into text data, means for receiving video data, means for extracting character information from the video data, means for translating the text data and the character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for capturing a user's facial expression and analyzing the emotion, and means for adjusting the system operation based on the analyzed emotion. This makes it possible not only to translate voice and video data in real time, but also to analyze the user's emotion and provide appropriate feedback.

[1342] "Audio data" is data that represents an audio signal in digital form.

[1343] "Text data" is character string data extracted from audio data or video data.

[1344] "Video data" refers to video data captured by a device such as a camera.

[1345] "Text information" refers to information such as characters and symbols extracted from video data.

[1346] "Translation" is the process of converting text data expressed in one language into another language.

[1347] "Distribution" means transmitting the generated video to viewers in real time via a network.

[1348] "User's facial expression" refers to the facial expression of the user captured by the camera.

[1349] "Analyzing emotions" refers to a process of identifying the emotions felt by the user based on the user's facial expression data.

[1350] "Adjusting the system's behavior" means changing the system's functions and behavior based on the analyzed emotional information.

[1351] System Overview

[1352] This invention is a system that translates audio and video content in real time during online classes and lectures, and also recognizes and provides feedback to users based on their emotions. The system consists of the following main components:

[1353] Voice Recognition Module

[1354] Character Recognition Module

[1355] Translation Module

[1356] Video generation and distribution module

[1357] Emotion Engine

[1358] Voice Recognition Module

[1359] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data is converted into a format such as WAV or MP3 and sent to the server. The server passes the received audio data to a speech recognition module, which converts it into text data. For example, if a user says, "Today I will talk about the law of conservation of energy," it will be converted into text data.

[1360] Character Recognition Module

[1361] When a user is teaching a lesson while writing on a whiteboard or blackboard, the device's camera captures the video. The captured video data is also sent to the server, and the server passes the received video data to a character recognition module (OCR). The character recognition module extracts character information from the video data. For example, if the string "E = mc²" is written on the blackboard, it will be extracted as text data.

[1362] Translation Module

[1363] The server passes the text data extracted from the speech recognition module and character recognition module to the translation module, which translates the text data into the specified language. For example, "Today we will talk about the law of conservation of energy" is translated into English as "Today we will discuss the law of conservation of energy." The mathematical formula "E = mc²" is kept as is.

[1364] Video generation and distribution module

[1365] The server generates a new video based on the translated text data. This new video is overlaid with subtitles of the translated text. The generated video is then distributed to viewers in real time. Viewers receive this video on their own devices and display it on their screens.

[1366] Emotion Engine

[1367] The user's device uses a built-in camera to capture the user's facial expressions. The captured video data is sent to a server, which then passes it to an emotion engine to analyze the user's facial expressions. The emotion engine recognizes the user's emotions and feeds that information back to the system. Based on this feedback, the system can reflect the information in translations and video generation, if necessary. For example, if the user has a confused expression, the system can respond by providing additional explanatory text.

[1368] Specific examples

[1369] For example, suppose a user is taking an online class in Japan. During this class, the device captures the instructor's voice and the contents of the blackboard and sends them to the server. The server converts the voice data into text and performs character recognition on the mathematical formulas and words written on the blackboard. This data is translated into English by the translation module, and then the video generation and distribution module generates a video with English subtitles and distributes it in real time to overseas students. The overseas students can then take the class with English subtitles on their own devices. Furthermore, the system captures the user's facial expressions in real time and analyzes their emotions with an emotion engine. For example, if the system determines that the user is confused, it provides additional supplementary explanations. In this way, high-quality education and lecture content can be provided worldwide, transcending language barriers and deepening users' understanding.

[1370] Example prompts for generative AI models

[1371] Example prompt 1:

[1372] "Please translate the Japanese audio data into English and distribute it with English subtitles."

[1373] Example prompt 2:

[1374] "Please recognize the mathematical formulas written on the blackboard and add them directly to the English subtitles."

[1375] Example prompt 3:

[1376] "Analyze the user's facial expression and display additional explanatory text if confused."

[1377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1378] Step 1:

[1379] Terminal

[1380] When a user starts an online class or lecture, the device captures audio in real time using the built-in microphone. The captured audio data (input) is converted into a format such as WAV or MP3 (data processing) and then sent to the server (output).

[1381] Specific behavior:

[1382] When a user starts speaking, "Today I'm going to talk about the law of conservation of energy," the audio is captured by the device's microphone. The captured audio data is converted into WAV format and sent from the device to the server.

[1383] Step 2:

[1384] server

[1385] The server passes the voice data (input) received from the terminal to the voice recognition module. This voice recognition module converts the voice data into character string data (text data) (data conversion). The character string data (output) is stored within the server.

[1386] Specific behavior:

[1387] When the server inputs the received voice data into the voice recognition module, the voice data "Today we will talk about the law of conservation of energy" is converted into text "Today we will talk about the law of conservation of energy."

[1388] Step 3:

[1389] Terminal

[1390] When a user is teaching a class while writing on a whiteboard or blackboard, the device camera captures this video. The captured video data (input) is sent to the server (output).

[1391] Specific behavior:

[1392] When a user writes "E = mc²" on the blackboard, the device's camera captures the image and the image data is sent to the server.

[1393] Step 4:

[1394] server

[1395] The server passes the received video data (input) to a character recognition module (OCR). This character recognition module extracts character information from the video data (data extraction). The extracted character information (output) is stored within the server.

[1396] Specific behavior:

[1397] When the video data received by the server is input into the character recognition module, the character information "E = mc²" is extracted and saved as text data.

[1398] Step 5:

[1399] server

[1400] The server passes the text data (input) obtained from the speech recognition module and character recognition module to the translation module. The translation module translates this data into the desired language (data translation). The translated text data (output) is stored within the server.

[1401] Specific behavior:

[1402] The text data "Today we will talk about the law of conservation of energy" is translated into "Today we will discuss the law of conservation of energy", and the text data "E = mc²" is left as is.

[1403] Step 6:

[1404] server

[1405] The server uses a video generation and distribution module to generate new video based on the translated text data (input). This module overlays the translated text data as subtitles (data generation). The generated video (output) is distributed to viewers in real time.

[1406] Specific behavior:

[1407] The server generates a video with an overlay of the subtitle "Today we will discuss the law of conservation of energy" and distributes it to viewers in real time.

[1408] Step 7:

[1409] Terminal

[1410] The user's device captures the user's facial expressions using a built-in camera, and the captured video data (input) is sent to the server (output).

[1411] Specific behavior:

[1412] If the user makes a confused expression, the expression is captured by the device's camera and the video data is sent to the server.

[1413] Step 8:

[1414] server

[1415] The server passes the received video data (input) to the emotion engine, which analyzes the user's facial expressions and recognizes their emotions (data analysis). The analyzed emotions (output) are used to adjust the system's behavior.

[1416] Specific behavior:

[1417] When the server inputs the captured user's facial expressions into an emotion engine, the emotion of confusion is recognized, and the system responds by generating and delivering additional explanatory text.

[1418] (Application example 2)

[1419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1420] Conventional online classes and lecture systems have had problems with real-time multilingual translation and providing feedback based on the user's level of understanding. Specifically, it is difficult to ensure that students and audience members who speak different languages ​​can instantly understand the content of the lecture. They also lack the functionality to supplement the content of the lecture based on the user's facial expressions and emotions. This has led to issues such as a decline in the quality of the classes and lectures and a decrease in the user's learning effectiveness.

[1421] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1422] In this invention, the server includes means for receiving audio data of a lesson or lecture, means for converting the audio data into text data, means for receiving video data of a whiteboard or blackboard, means for extracting character information from the video data, means for translating the text data and character information into a desired language, means for generating a video based on the translated text data and the translated character information, means for distributing the generated video, means for receiving user facial expression data and analyzing the emotions, and means for providing additional explanations based on the analyzed emotion data. This makes it possible for users who speak different languages ​​to understand the content of the lesson in real time, and furthermore, by providing feedback based on the user's emotions, it is possible to improve the quality of the lesson or lecture and increase the user's learning effectiveness.

[1423] "Means for receiving audio data" refers to a device or function for capturing the audio of a class or lecture, converting it into digital data, and transmitting it to a server.

[1424] The "means for converting voice data into text data" refers to a voice recognition technology for analyzing received voice data and extracting its contents as text information.

[1425] "Means for receiving video data" refers to a device or function that captures what is written on a whiteboard or blackboard using a camera or other device and transmits it to a server as digital data.

[1426] The "means for extracting character information from video data" is a means for analyzing captured video data, recognizing characters therein, and converting them into text data.

[1427] "Means for translating text data and character information into a desired language" refers to a translation technology for converting text data obtained by speech recognition or character recognition into a language desired by the user.

[1428] "Means for generating video based on translated text data and translated character information" refers to video generation technology for creating new video based on translated text data and adding subtitles, etc. to that video.

[1429] The "means for distributing the generated video" refers to a network and software for distributing the generated video data to users in real time.

[1430] The "means for receiving the user's facial expression data and analyzing the emotions" is a technology for capturing the user's facial image in real time, analyzing the emotions, and acquiring them as data.

[1431] The "means for providing additional explanations based on analyzed emotion data" is an automatic response system that provides supplementary explanations of the content of a class or lecture based on the analyzed emotion of the user.

[1432] This invention relates to a system that performs real-time translation and sentiment analysis for online classes and lectures. This system consists of the following steps and components:

[1433] First, the server has a means for receiving the audio data of the class or lecture. This includes devices and functions that capture audio using a microphone installed on the terminal, convert the audio data into a digital format, and send it to the server. The server then converts the received audio data into text data using speech recognition technology.

[1434] Next, the server is equipped with a means for receiving video data. This includes a device or function that captures what is written on the whiteboard or blackboard with a camera, converts the video data into a digital format, and transmits it to the server. The server then extracts text information from the received video data using optical character recognition (OCR) technology.

[1435] Furthermore, the server translates the text data obtained by speech recognition and character recognition into the desired language using multilingual translation technology, such as the Google Translate API.

[1436] The server generates a new video based on the translated text data and the translated character information. This video generation includes technology to overlay the text data as subtitles. The generated video is delivered to the user's device in real time.

[1437] Furthermore, the server is equipped with a means for receiving the user's facial expression data and analyzing the emotions. The device uses a camera to capture the user's face and transmits the facial expression data to the server. The server then uses an emotion recognition algorithm to analyze the user's emotions in real time. For example, facial recognition technology and deep learning models can be used for emotion recognition.

[1438] Based on the analyzed emotional data, the server provides appropriate feedback. For example, if it determines that the user is confused, it provides additional explanations via voice or text. This feedback deepens the user's understanding and improves the quality of lessons or lectures.

[1439] A specific example of this application is a language learning platform. Using this system, users can understand lectures from instructors in multiple languages ​​in real time. Furthermore, if users have questions, additional explanations are automatically provided, improving learning effectiveness. This system is implemented using technologies such as Google Translate API, OpenCV, SpeechRecognition, and Pyttsx3.

[1440] Example prompt sentence:

[1441] "Please translate what is being said in Japanese into English and display it as subtitles on the screen. Additionally, please analyze the user's facial expressions and provide additional explanations if they are confused."

[1442] In this way, we provide a system that can perform translation and emotion analysis in real time to support user understanding.

[1443] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1444] Step 1:

[1445] Capture audio data

[1446] The device captures the audio data of a class or lecture using a microphone. The input is the teacher's or lecturer's audio data, and the output is audio data in a digital format (e.g., WAV, MP3). This digital data is sent to a server.

[1447] Step 2:

[1448] Speech-to-text conversion

[1449] The server passes the received audio data to a speech recognition module. The input is digital audio data, and the output is the corresponding text data. This conversion is done using a speech recognition library such as SpeechRecognition.

[1450] Step 3:

[1451] Video data capture

[1452] The device captures video data of the whiteboard or blackboard with a camera. The input is what the teacher or lecturer has written, and the output is digital video data. This digital data is sent to the server.

[1453] Step 4:

[1454] Extracting text information from video

[1455] The server passes the received video data to a character recognition module. The input is digital video data, and the output is extracted text information. This conversion is performed using optical character recognition (OCR) technology.

[1456] Step 5:

[1457] Text data translation

[1458] The server passes the text data from the speech recognition module and the character information from the character recognition module to the translation module. The input is the text data and character information, and the output is the text data translated into the desired language. This translation is performed using multilingual translation technologies such as DeepL or Google Translate API.

[1459] Step 6:

[1460] Video generation

[1461] The server generates new videos based on the translated text data. The input is the translated text data and character information, and the output is video data with subtitles added. Video generation technology is used for this generation.

[1462] Step 7:

[1463] Video distribution

[1464] The server distributes the generated video data to viewers in real time. The input is the generated video data, and the output is the video displayed on the user's device.

[1465] Step 8:

[1466] Capturing facial expression data

[1467] The device captures the user's facial expression with a camera. The input is the user's facial image data, and the output is the digital data. This digital data is sent to the server.

[1468] Step 9:

[1469] Emotion Analysis

[1470] The server passes the received facial expression data to the emotion engine. The input is digital facial image data, and the output is analyzed emotion data. This analysis is performed using a deep learning model.

[1471] Step 10:

[1472] Providing feedback

[1473] The server generates additional explanations as needed based on the analyzed emotion data. The input is emotion data, and the output is additional feedback information (audio or text). For example, if the server determines that the user is confused, it generates supplementary explanations and provides them in audio or text.

[1474] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1476] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1477] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1478] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1479] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1480] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1481] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1482] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1483] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1484] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1485] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1486] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1487] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1488] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1489] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1490] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1491] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1492] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1493] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1494] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1495] The following is further disclosed regarding the above embodiment.

[1496] (Claim 1)

[1497] a means for receiving audio data of a class or lecture;

[1498] means for converting the voice data into text data;

[1499] means for receiving video data of a whiteboard or blackboard;

[1500] means for extracting character information from the video data;

[1501] means for translating the text data and the character information into a desired language;

[1502] means for generating an image based on the translated text data and the translated character information;

[1503] A system including means for distributing the generated video.

[1504] (Claim 2)

[1505] 10. The system of claim 1, wherein the audio data is received in real time.

[1506] (Claim 3)

[1507] 10. The system of claim 1, wherein character information is extracted from the video data using optical character recognition technology.

[1508] "Example 1"

[1509] (Claim 1)

[1510] a means for receiving audio data of a class or lecture;

[1511] means for converting the voice data into text data;

[1512] means for receiving video data of a whiteboard or blackboard;

[1513] means for extracting character information from the video data;

[1514] means for translating the text data and the character information into a desired language;

[1515] means for generating an image based on the translated text data and the translated character information;

[1516] A system including means for distributing the generated video in real time.

[1517] (Claim 2)

[1518] 10. The system of claim 1, further comprising means for receiving said audio data in real time and converting said audio data into a format that is easy to process.

[1519] (Claim 3)

[1520] 2. The system according to claim 1, further comprising means for extracting character information from said video data using optical character recognition technology and translating said character information into a desired language.

[1521] (Claim 4)

[1522] 2. The system according to claim 1, further comprising means for generating a video in which the translated text data and the translated character information are overlaid as subtitle information.

[1523] (Claim 5)

[1524] 10. The system of claim 1, further comprising means for generating a video file containing the subtitle information and delivering the video file to a viewer using HTTP Live Streaming Protocol.

[1525] "Application Example 1"

[1526] (Claim 1)

[1527] a means for receiving audio data of a class or lecture;

[1528] means for converting the voice data into text data;

[1529] means for receiving video data of a whiteboard or blackboard;

[1530] means for extracting character information from the video data using optical character recognition technology;

[1531] means for translating the text data and the character information into a desired language;

[1532] means for generating an image based on the translated text data and the translated character information;

[1533] means for delivering the generated video to a user terminal in real time;

[1534] The system includes means for displaying the generated image on a user's terminal.

[1535] (Claim 2)

[1536] 10. The system of claim 1, wherein the audio and video data is received and translated in real time.

[1537] (Claim 3)

[1538] The system according to claim 1, wherein the system uses an application installed on a user terminal such as a smartphone or a head-mounted display.

[1539] "Example 2: Combining Emotion Engines"

[1540] (Claim 1)

[1541] means for receiving audio data;

[1542] means for converting the voice data into text data;

[1543] means for receiving video data;

[1544] means for extracting character information from the video data;

[1545] means for translating the text data and the character information into a desired language;

[1546] means for generating an image based on the translated text data and the translated character information;

[1547] means for distributing the generated video;

[1548] means for capturing a user's facial expression and analyzing the emotion;

[1549] a means for adjusting the behavior of the system based on the analyzed emotions; and

[1550] A system including:

[1551] (Claim 2)

[1552] 10. The system of claim 1, wherein the audio data is received in real time.

[1553] (Claim 3)

[1554] 10. The system of claim 1, wherein character information is extracted from the video data using optical character recognition technology.

[1555] "Application example 2 when combining emotion engines"

[1556] (Claim 1)

[1557] a means for receiving audio data of a class or lecture;

[1558] means for converting the voice data into text data;

[1559] means for receiving video data of a whiteboard or blackboard;

[1560] means for extracting character information from the video data;

[1561] means for translating the text data and the character information into a desired language;

[1562] means for generating an image based on the translated text data and the translated character information;

[1563] means for distributing the generated video;

[1564] means for receiving facial expression data of a user and analyzing the emotion;

[1565] means for providing additional explanation based on the analyzed emotion data;

[1566] A system including:

[1567] (Claim 2)

[1568] 10. The system of claim 1, wherein the audio data is received in real time.

[1569] (Claim 3)

[1570] 10. The system of claim 1, wherein character information is extracted from the video data using optical character recognition technology. [Explanation of symbols]

[1571] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving audio data of a class or lecture; means for converting the voice data into text data; means for receiving video data of a whiteboard or blackboard; means for extracting character information from the video data; means for translating the text data and the character information into a desired language; means for generating an image based on the translated text data and the translated character information; A system including means for distributing the generated video.

2. 10. The system of claim 1, wherein the audio data is received in real time.

3. The system of claim 1, wherein character information is extracted from the video data using optical character recognition techniques.

Images