system

Abstract

Provide a system. [Solution] means for extracting audio data from video data in the terminal; means for transmitting the extracted voice data to a server; a means for performing speech recognition using the received speech data in the server and generating corresponding text data; means for transmitting the generated text data to a terminal; means for displaying the received text data in a form corresponding to the video data in the terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the amount of video content has rapidly increased, and much of it is provided with audio. However, audio-only information is difficult to access for people with hearing impairments and those who work in noisy environments. Language barriers also exist, making it difficult for people who speak different languages ​​to understand the same content. To address these issues, a system for automatically generating subtitles is needed, but generating subtitles in real time poses many technical challenges. The objective of this invention is to provide an efficient system for solving these challenges. [Means for solving the problem]

[0005] The present invention provides a system including: a means for extracting audio data from video data at a terminal; a means for transmitting the extracted audio data to a server; a means for performing speech recognition at the server using the received audio data and generating corresponding text data; a means for transmitting the generated text data to the terminal; and a means for displaying the received text data at the terminal in a form corresponding to the video data. This system improves the accessibility of video content by converting audio to text in real time and formatting it into an appropriate subtitle format for display. Furthermore, by including a means for dividing the extracted audio data at the terminal into multiple chunks and sequentially transmitting them to the server, and a means for assigning a timestamp to the generated text data at the server and converting it into a subtitle format, both real-time performance and accuracy can be achieved.

[0006] A "terminal" is a device operated by a user, and in this context refers primarily to mobile devices such as smartphones and tablets.

[0007] "Video data" refers to digital data containing visual information of video content.

[0008] "Audio data" refers to digital data of sound extracted from an audio track included in video data.

[0009] "Extraction" refers to the operation of extracting specific information (here, audio data) from video data.

[0010] "Server" refers to a computer system that processes, manages, and distributes data over a network.

[0011] "Speech recognition" refers to the technology of analyzing voice data and outputting the content as text data.

[0012] "Text data" refers to digital data containing text information generated by speech recognition.

[0013] "Send" refers to the operation of transferring data between terminals and servers via a communication line.

[0014] "Subtitles" are text data that are displayed in correspondence with video data, and usually serve to visually supplement dialogue and explanations.

[0015] A "chunk" refers to a small unit of data that is divided into pieces to make it easier to handle.

[0016] A "timestamp" refers to the time information when a specific event (in this case, speech recognition or text generation) occurred.

[0017] "Formatting" refers to the act of converting or adjusting data into a specific format or form.

[0018] "Real-time" refers to data processing and display occurring immediately, without delay. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention uses a system that combines a terminal and a server to transcribe the audio in a video in real time and display it as subtitles. The specific operation of this system is described below.

[0041] System Overview

[0042] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[0043] Program structure and operation

[0044] User operations

[0045] When a user selects a video using a video playback app such as the YouTube app on their device and starts playing it, the system works in the background to extract audio and generate subtitles.

[0046] Terminal side processing

[0047] The device extracts the video's audio track in real time as soon as the video starts playing, using a MediaPlayer object to play the audio and an AudioRecord object to capture the audio data.

[0048] The extracted audio data is divided into chunks of a predetermined size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[0049] Server-side processing

[0050] The server saves the received voice data in a temporary file and sends it to a speech recognition service such as Google® Cloud Speech-to-Text API. The text data generated by the speech recognition is returned to the server with a timestamp.

[0051] The server parses the returned text data and formats it into a subtitle format, usually .srt or WebVTT, and sends the formatted subtitle data to the device.

[0052] Subtitle display on the device

[0053] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[0054] Specific examples

[0055] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[0056] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[0057] 1

[0058] 00:00:01,000 --> 00:00:04,000

[0059] Hello, this is an introductory text for Python.

[0060] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[0061] Advantages of the invention

[0062] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[0066] Step 2:

[0067] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[0068] Step 3:

[0069] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[0070] Step 4:

[0071] The server saves the received voice data as a temporary file, which is then ready to be sent to the voice recognition server.

[0072] Step 5:

[0073] The server calls a speech recognition service, such as the Google Cloud Speech-to-Text API, and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[0074] Step 6:

[0075] The server analyzes the received text data and timestamp information and formats it into .srt or WebVTT format. Specifically, it defines the text and its start and end times.

[0076] Step 7:

[0077] The server sends the formatted subtitle data to the terminal as an HTTP response.

[0078] Step 8:

[0079] The device analyzes the subtitle data received and processes it to display it in a TextView object. The subtitle display is implemented using Handlers and Runnables so that it is updated in real time based on timestamps.

[0080] Step 9:

[0081] When a user watches a video, the device overlays subtitles onto the video in real time, allowing the user to understand the video content not only audibly but also visually.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Video content is consumed daily, but people with hearing impairments and those in noisy environments often find it difficult to understand the audio. Furthermore, when watching videos in foreign languages, there is a demand for a function that displays audio as text data. However, existing systems lack real-time capabilities and are limited in their use due to limitations on the devices and services they can be used with, resulting in a lack of flexibility. Furthermore, delays in the display of subtitles can degrade the user experience.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes means for performing speech recognition processing and generating corresponding character data, means for formatting the generated character data into a subtitle format and attaching a timestamp, and means for transmitting the formatted subtitle data to the terminal, thereby enabling accurate subtitle display in real time and improving the user experience.

[0087] A "terminal" is a device that plays video, extracts audio, and sends it to a server.

[0088] "Audio signal" refers to data captured in real time from audio data within a video.

[0089] "Real time" means that processing, from audio extraction to subtitle display, is carried out almost simultaneously.

[0090] The "predetermined size" is the unit by which the audio signal is divided into a specified interval (for example, every second).

[0091] "Asynchronous" means that audio data is transmitted in real time, but each transmission is carried out independently.

[0092] The "server" is a device that receives an audio signal, performs voice recognition processing, generates subtitle data, and transmits it to a terminal.

[0093] "Speech recognition processing" is a technology that converts voice signals into text data.

[0094] "Character data" is text-format data generated by voice recognition.

[0095] A "subtitle format" is a specific format (e.g., .srt or WebVTT) used to display audio in a video as text.

[0096] A "timestamp" is information that is added to character data and indicates the exact timing of display.

[0097] "Overlay display" is a method of displaying content over the video playback screen.

[0098] The present invention uses a system that combines a terminal and a server to transcribe audio from a video in real time and display the transcribed audio as subtitles. An embodiment of this system will be described in detail below.

[0099] System Overview

[0100] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[0101] User operations

[0102] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[0103] Terminal side processing

[0104] The device extracts the audio signal of the video in real time as soon as the video starts playing. To do this, it uses a MediaPlayer object to play the audio and an AudioRecord object to capture the audio signal. The captured audio signal is divided into chunks of a specified size and sent to the server asynchronously. Data is sent to the server every second.

[0105] Server-side processing

[0106] The server saves the received audio signal to a temporary file and sends the data to a speech recognition service (e.g., Google Cloud Speech-to-Text API). The text data generated by speech recognition is returned to the server with a timestamp. The server parses the returned text data and formats it into a subtitle format (e.g., .srt, WebVTT). The formatted subtitle data is then sent to the device.

[0107] Subtitle display on the device

[0108] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, allowing the user to view the subtitles in real time.

[0109] Specific examples

[0110] For example, consider the case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio signal and generates text data such as "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file as follows:

[0111] 1

[0112] 00:00:01,000 --> 00:00:04,000

[0113] Hello, this is an introductory text for Python.

[0114] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[0115] Prompt Sentence Examples

[0116] Prompt for explaining a program that uses generative AI models to generate real-time subtitles:

[0117] "Please tell me the process flow of a program that captures the audio of a video being played on the YouTube app and generates subtitles in real time using the Google Cloud Speech-to-Text API."

[0118] Advantages of the invention

[0119] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The user plays a video on their device. The input is the user selecting the video they want to play, and the output is the video starting to play on the device. Specifically, the user selects a video using the YouTube app or similar and presses the play button.

[0123] Step 2:

[0124] When the device starts playing the video, it also starts playing the audio signal using a MediaPlayer object. The input is the video file, and the output is the audio signal played in real time. Specifically, the MediaPlayer processes the audio signal and plays the audio.

[0125] Step 3:

[0126] The device uses an AudioRecord object to capture audio signals in real time. The input is the audio signal played by the MediaPlayer, and the output is the captured audio signal. Specifically, AudioRecord captures the audio signal and stores it in memory.

[0127] Step 4:

[0128] The device divides the captured audio signal into chunks of a specified size and sends them asynchronously to the server. The input is the captured audio signal, and the output is the audio signal data divided into chunks of a specified size (e.g., every second). Specifically, the audio signal is divided into chunks every second and sent to the server over the network.

[0129] Step 5:

[0130] The server saves the received audio signal in a temporary file. The input is the audio signal data sent from the terminal, and the output is the audio data saved in the temporary file. Specifically, the server saves the received audio signal as temp.wav.

[0131] Step 6:

[0132] The server sends the audio signal stored in a temporary file to the speech recognition service. The input is the temporary file of the stored audio signal, and the output is a request to the speech recognition service. Specifically, the server sends the audio file to the Google Cloud Speech-to-Text API and requests text data.

[0133] Step 7:

[0134] The speech recognition service analyzes the audio signal and generates corresponding text data. The input is the transmitted audio file, and the output is the generated text data. Specifically, the speech recognition service analyzes the audio signal and returns text data such as "Hello, this is an introductory text for Python."

[0135] Step 8:

[0136] The server adds a timestamp to the generated text data and formats it into subtitle format. The input is the text data returned from the speech recognition service, and the output is subtitle format data with a timestamp. Specifically, the text data is formatted into the following .srt format:

[0137] 1

[0138] 00:00:01,000 --> 00:00:04,000

[0139] Hello, this is an introductory text for Python.

[0140] Step 9:

[0141] The server sends the formatted subtitle data to the terminal. The input is time-stamped subtitle data, and the output is an API response to the terminal. Specifically, the server sends the formatted subtitle data to the terminal as an HTTP response.

[0142] Step 10:

[0143] The device analyzes the subtitle data received from the server and displays it as an overlay on the video playback screen. The input is the subtitle data sent from the server, and the output is real-time subtitles displayed on the video screen. Specifically, the device parses the received .srt file and displays the subtitles based on the timestamp. The user can view the subtitles in real time.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Current video streaming services do not adequately provide the functionality to display subtitles of audio content in real time while watching. This makes it difficult for the hearing impaired and users in multilingual environments to understand video content. In addition, the functionality available varies depending on the device, making it difficult to provide a consistent viewing experience for all users.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes means for performing speech recognition using received voice data and generating corresponding text data, means for assigning a timestamp to the generated text data, and means for converting the time-stamped text data into a subtitle format, thereby enabling subtitles to be generated and displayed in real time on a variety of playback devices.

[0149] A "terminal" is a device that extracts audio data, communicates with a server to receive text data, and displays the text data in a format that corresponds to the video data.

[0150] "Video data" refers to data of video content that a user views.

[0151] "Audio data" refers to data that indicates the audio portion included in the video data.

[0152] A "server" is a computer system that receives voice data sent from a terminal and performs voice recognition and generates text data.

[0153] "Speech recognition" is a process of converting received voice data into text data.

[0154] "Text data" is character string information generated by speech recognition.

[0155] A "timestamp" is time information added to text data, and indicates the timing of displaying subtitles.

[0156] The "subtitle format" is a data format for appropriately formatting the generated text data and displaying it in correspondence with the video data.

[0157] A "video distribution service" is an online service that distributes video content over the Internet.

[0158] "Real-time subtitle generation" is a function that instantly converts audio data into text data and displays it while a video is playing.

[0159] "A wide variety of playback devices" refers to various types of devices, such as smartphones, smart glasses, and head-mounted displays.

[0160] The present invention provides a system for generating and displaying subtitles in real time for video distribution services. This system is configured by combining a terminal, a server, and a voice recognition service.

[0161] System Configuration

[0162] 1. Device:

[0163] The hardware used includes smartphones, smart glasses, and head-mounted displays.

[0164] Audio data is extracted from the video data simultaneously with the video playback.

[0165] To capture audio data, in the case of a smartphone, MediaPlayer (ANDROID (registered trademark)) or AVPlayer (iOS) is used, and AudioRecord (Android) or AVAudioRecorder (iOS) is used to record the audio data.

[0166] 2. Server:

[0167] Receives voice data sent from the terminal and performs voice recognition.

[0168] The Google Cloud Speech-to-Text API is used as the voice recognition service.

[0169] The text data generated by speech recognition is time-stamped and formatted into .srt or WebVTT format.

[0170] 3. Data flow:

[0171] When the device starts playing the video, it extracts the audio data and divides it into chunks of a specified size (e.g., 1 second).

[0172] The device asynchronously sends this to the server, which temporarily stores the received audio data and sends it to the Google Cloud Speech-to-Text API.

[0173] It receives the text data returned as a result of speech recognition, adds a timestamp to it, and formats it into subtitle format.

[0174] The formatted subtitle data is sent to the terminal, which receives it and displays it as subtitles.

[0175] Specific examples

[0176] Consider a specific example where a user is watching a movie on a smartphone using a video streaming service. When the user starts playing the video, the device captures the audio data and sends it to a server. The server then sends the received audio data to a speech recognition service, which generates text data such as "Once upon a time...". The generated text data is then given a timestamp and formatted in .srt format. This subtitle data is then sent to the device and displayed on the video screen in real time.

[0177] Prompt Sentence Examples

[0178] Develop an application that captures the audio track while a user is watching a movie on a video streaming service on their smartphone, converts it into text using the Google Cloud Speech-to-Text API, and displays it as real-time subtitles.

[0179] This system makes it possible to generate and display subtitles in real time on a variety of devices for video streaming services, thereby providing a consistent, high-quality viewing experience for users with hearing impairments and those in multilingual environments.

[0180] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0181] Step 1:

[0182] A user launches a video streaming service application on a device (smartphone, smart glasses, head-mounted display) and starts playing a video. At this point, the audio data extraction process begins along with the playback of the video data. Specifically, the device uses MediaPlayer (Android) or AVPlayer (iOS) to play the audio track of the video data in real time, and simultaneously captures the audio data using AudioRecord (Android) or AVAudioRecorder (iOS).

[0183] Input: Video playback start command, video data

[0184] Output: Extracted audio data

[0185] Step 2:

[0186] The extracted audio data is divided into chunks of a specified size (e.g., 1 second). The device sends this audio data to the server asynchronously. The division and transmission of the audio data is performed continuously, and continues without interruption while the user is watching the video. This maintains real-time performance.

[0187] Input: Extracted audio data

[0188] Output: Audio data chunks divided into predetermined sizes

[0189] Step 3:

[0190] The server temporarily stores the received audio data chunks and sends them to a speech recognition service such as the Google Cloud Speech-to-Text API, which converts the audio data into text data.

[0191] Input: Audio data chunks divided into predetermined sizes

[0192] Output: Generated text data

[0193] Step 4:

[0194] The server appends a timestamp to the text data returned by the speech recognition service and formats it into a subtitle format such as .srt or WebVTT. During this formatting process, the text data and the timestamp are formatted accordingly.

[0195] Input: Generated text data

[0196] Output: Subtitle data with timestamps

[0197] Step 5:

[0198] The server sends the formatted subtitle data to the device. The device receives this data and displays it as an overlay on the video playback screen. The display timing is based on the timestamp in real time, allowing the user to visually understand the audio content.

[0199] Input: Formatted subtitle data

[0200] Output: Display subtitles on the video playback screen

[0201] Through these processing steps, the video streaming service application can transcribe the audio of the video being watched by the user in real time and display it as subtitles, making it easier for users with hearing impairments and those in multilingual environments to understand the video content.

[0202] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0203] This invention combines a system that transcribes the audio of video content in real time and displays it as subtitles with an emotion engine, recognizing the user's emotional state and displaying it together with the subtitle data, thereby providing video viewers with richer information.

[0204] System Overview

[0205] This system consists of a terminal and a server. The terminal is operated by the user and is responsible for playing videos, extracting audio data, and recognizing emotions. The server, on the other hand, is responsible for voice recognition and generating subtitles. Furthermore, an emotion engine recognizes the user's emotions and displays the results along with the subtitle data.

[0206] Program structure and operation

[0207] User operations

[0208] The user selects a video using a video playback app on their device and starts playing it, while the system works in the background, extracting audio and recognizing emotions.

[0209] Terminal side processing

[0210] When the device starts playing the video, it uses the MediaPlayer object to play the video's audio track, and simultaneously captures the audio data in real time using the AudioRecord object. The captured audio data is divided into chunks of a specified size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[0211] The emotion engine also analyzes the user's facial expressions and tone of voice using the user's camera footage and audio signals. The analyzed emotion data is updated in real time.

[0212] Server-side processing

[0213] The server saves the received audio data to a temporary file, then calls a speech recognition service such as the Google Cloud Speech-to-Text API and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[0214] The server parses the returned text data and formats it into subtitles, which are then sent to the device.

[0215] Subtitle display on the device

[0216] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp. The analyzed emotional data is also displayed at the same time. This allows the user to understand not only the audio information but also the emotional state.

[0217] Specific examples

[0218] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[0219] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[0220] 1

[0221] 00:00:01,000 --> 00:00:04,000

[0222] Hello, this is an introductory text for Python.

[0223] At the same time, the emotion engine analyzes the user's facial expressions and tone of voice to generate emotion data such as "excitement" or "concentration." For example, the following display is generated:

[0224] 1

[0225] 00:00:01,000 --> 00:00:04,000

[0226] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[0227] The device receives this subtitle data and emotion data and displays them on the video playback screen, allowing the user to view the emotion information along with the subtitles in real time.

[0228] Advantages of the invention

[0229] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, analyzing and displaying the user's emotional information can further enrich the viewing experience. Another major advantage is the simple system configuration, which combines ease of implementation with high accuracy by utilizing existing cloud services.

[0230] The processing flow will be explained below.

[0231] Step 1:

[0232] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[0233] Step 2:

[0234] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[0235] Step 3:

[0236] To perform real-time emotion recognition, the device captures the user's face with a camera and the user's voice tone with a microphone, and these data are analyzed by the emotion engine.

[0237] Step 4:

[0238] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[0239] Step 5:

[0240] The server saves the received voice data as a temporary file and sends it to a speech recognition service such as Google Cloud Speech-to-Text API, which converts the voice data into text data and returns a response with a timestamp.

[0241] Step 6:

[0242] The server analyzes the received text data and its timestamp information and formats it into subtitle format (for example, .srt format).

[0243] Step 7:

[0244] The server sends the formatted subtitle data to the terminal as an HTTP response.

[0245] Step 8:

[0246] The device analyzes the received text data and processes it to overlay it on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[0247] Step 9:

[0248] The device displays the analyzed emotion data along with the text data, which includes the user's emotional state (e.g., sad, happy, etc.) analyzed by the emotion engine.

[0249] Step 10:

[0250] Users can view their own emotional state along with subtitles while watching videos in real time, improving the viewing experience and providing emotional feedback.

[0251] Example 2

[0252] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0253] Conventional subtitling systems primarily convert audio data into text data for display, and are unable to reflect the user's emotional information. Furthermore, they lack the ingenuity to further enrich the user's viewing experience, limiting the information they can provide to viewers. As a result, there is a need for a method to further deepen the understanding of video content, especially for the hearing impaired and in environments requiring multilingual support.

[0254] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for extracting audio data from video data at the terminal; means for transmitting the extracted audio data to the server; means for performing speech recognition using the audio data received at the server and generating corresponding text data; means for transmitting the generated text data to the terminal; means for displaying the received text data at the terminal in a form corresponding to the video data; means for analyzing the user's facial expression and tone of voice at the terminal and generating emotional data of the user; and means for displaying the generated emotional data together with the text data in a form corresponding to the video data. This further enriches the user's viewing experience and enables a deeper understanding of video content, particularly for the hearing impaired or in an environment requiring multilingual support.

[0255] A "terminal" is a device that is operated by a user and has the functions of playing back video data, extracting audio data, and generating emotion data.

[0256] "Video data" refers to data that includes visual information of moving image content, and is played back in synchronization with audio data.

[0257] "Audio data" refers to the audio included in the video data, and is sound information that can be heard by the user.

[0258] A "server" is a computing device that receives data sent from a terminal and performs voice recognition and data conversion.

[0259] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[0260] "Text data" refers to data containing character information obtained by voice recognition, and is displayed in correspondence with video data.

[0261] "Emotion data" is data that indicates the emotional state of the user, obtained by analyzing the user's facial expressions and tone of voice.

[0262] The "subtitle format" is a format in which a timestamp is added to text data and displayed in synchronization with video data.

[0263] "Platform" means a system that includes the underlying software and hardware for performing video playback, audio capture, and sentiment analysis.

[0264] This system transcribes the audio of video content in real time, displays it as subtitles, and simultaneously recognizes the user's emotional state and displays it together with the subtitle data. This system consists of a terminal operated by the user and a server that processes the data.

[0265] A device is a device on which a user selects a video using a video playback app and starts playback. The device is equipped with a MediaPlayer object, which plays the video's audio track, and simultaneously captures audio data in real time using an AudioRecord object. The captured audio data is stored in a buffer and then split into 1-second chunks and sent asynchronously to the server.

[0266] The device also comes equipped with an emotion engine that uses camera footage and microphone input to analyze facial expressions and vocal tone, generating emotion data that is updated in real time.

[0267] The server receives the audio data sent from the device and saves it as a temporary file. This saved audio data is then converted into text data using a speech recognition service such as the Google Cloud Speech-to-Text API. The text data is given a timestamp and is analyzed and formatted within the server. The formatted subtitle data is then sent back to the device.

[0268] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, and the analyzed emotional data is also displayed. This allows the user to understand not only the audio information but also the emotional state of the person.

[0269] For example, if a user plays a video titled "Introduction to Python," the device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio data and generates the text "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file and displayed as follows:

[0270] srt

[0271] 1

[0272] 00:00:01,000 --> 00:00:04,000

[0273] Hello, this is an introductory text for Python.

[0274] At the same time, the emotion engine analyzes the user's facial expressions and voice tone to generate emotion data such as "excitement" or "concentration." The device receives this subtitle data and emotion data and displays it in real time on the video playback screen. For example, it can be displayed as follows:

[0275] srt

[0276] 1

[0277] 00:00:01,000 --> 00:00:04,000

[0278] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[0279] An example of a prompt is, "Please explain the detailed process flow of a system that converts video audio into text in real time and displays it as subtitles, while also analyzing user emotions."

[0280] This invention allows users to enjoy subtitles in real time, deepening their understanding of video content, especially for the hearing impaired and in environments where multilingual support is required. Another major feature is that it can enrich the viewing experience by analyzing and displaying emotional information.

[0281] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0282] Step 1:

[0283] User plays a video

[0284] A user uses a video playback app to select a video and begin playing it.

[0285] Input: Select a video file and trigger playback.

[0286] Output: Video playback start signal.

[0287] Step 2:

[0288] Capture audio data

[0289] The device plays the audio track of the selected video using a MediaPlayer object, while simultaneously capturing the audio data in real time using an AudioRecord object.

[0290] Input: The audio track of the video file.

[0291] Output: Real-time captured audio data.

[0292] Specific operation: The captured audio data is temporarily stored in a buffer and divided into 1-second chunks.

[0293] Step 3:

[0294] Capturing Emotional Data

[0295] The device uses the user's camera footage and microphone input to analyze the user's facial expressions and voice tone using an emotion engine.

[0296] Input: User's camera video and audio input.

[0297] Output: Real-time analyzed emotion data.

[0298] Specific operation: By recognizing facial expressions from camera footage and analyzing the tone of voice from microphone input, an emotional state (such as "excitement" or "concentration") is generated.

[0299] Step 4:

[0300] Sending voice data to the server

[0301] The device asynchronously sends captured audio data to the server every second.

[0302] Input: Segmented audio data.

[0303] Output: Audio data sent to server completed.

[0304] Specific operation: The divided audio data is sent to the server using a protocol such as an HTTP request.

[0305] Step 5:

[0306] Saving audio data

[0307] The server saves the received audio data as a temporary file.

[0308] Input: Transmitted audio data.

[0309] Output: Audio data saved to a temporary file.

[0310] Specific operation: The audio data is saved as a file in WAV format or similar, and a timestamp is added to the file name.

[0311] Step 6:

[0312] Transcription of audio data

[0313] The server sends the stored audio data to the Google Cloud Speech-to-Text API and converts it into text data.

[0314] Input: Saved audio data file.

[0315] Output: Text data (with timestamp).

[0316] Specific operation: Calls the speech recognition service and converts the speech data into text format.

[0317] Step 7:

[0318] Subtitle data formatting

[0319] The server analyzes the converted text data and formats it into time-stamped subtitle data (such as .srt format).

[0320] Input: Text data generated by speech recognition.

[0321] Output: Formatted subtitle data.

[0322] Specific operation: Add appropriate line breaks and timestamps to the text data and format it into subtitle format.

[0323] Step 8:

[0324] Subtitle data transmission to device

[0325] The server sends the formatted subtitle data to the terminal.

[0326] Input: Formatted subtitle data.

[0327] Output: Subtitle data has been sent to the device.

[0328] Specific operation: Subtitle data is sent to the device using a protocol such as an HTTP request.

[0329] Step 9:

[0330] Displaying subtitle data and emotion data

[0331] The device analyzes the subtitle data received and the generated emotional data, and overlays it on the video playback screen in real time.

[0332] Input: Received subtitle data, parsed emotion data.

[0333] Output: Subtitles and emotions on the video playback screen.

[0334] Specific operation: Subtitle data and emotion data are synchronized based on timestamps and displayed in the appropriate position.

[0335] (Application example 2)

[0336] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] Conventional video playback systems have difficulty displaying subtitles in real time while a user is watching a video. Furthermore, they lack emotion recognition capabilities and are unable to display subtitles that take the user's emotional state into account, limiting the viewing experience. Real-time subtitle generation and emotional information display are particularly necessary for environments requiring support for the hearing impaired and multilingual support.

[0338] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for performing speech recognition and generating corresponding text data, means for assigning a timestamp to the generated text data and converting it into a subtitle format, and means for displaying emotion data generated in real time together with subtitles based on the timestamp. This makes it possible to generate subtitles in real time and display subtitles according to the user's emotional state.

[0339] A "terminal" is a device operated by a user that plays videos, extracts audio data, and recognizes emotions.

[0340] "Video data" refers to visual information contained in video content.

[0341] "Audio data" refers to the data of the audio track included in the video.

[0342] "Extraction" is the operation of extracting specific information (in this case, audio data) from a video.

[0343] "Transmit" means transferring data from one device to another device or server.

[0344] A "server" is a computer system that performs processes such as speech recognition and text generation.

[0345] "Speech recognition" is a technology that analyzes voice data and generates corresponding text data.

[0346] "Text data" is character information generated by voice recognition.

[0347] "Subtitle data" is text data to which a timestamp is added, and is displayed superimposed on video data.

[0348] A "timestamp" is time information that indicates a specific moment.

[0349] A "subtitle format" is a particular format in which text is displayed in video data.

[0350] "Facial expressions" are movements that show emotions and reactions on the user's face.

[0351] "Identification" is the act of analyzing video data and facial expression data to extract specific features and information.

[0352] An "emotional state" is information that describes the emotions a user is experiencing at a particular moment.

[0353] "Analysis" refers to examining data in detail to clarify its structure and meaning.

[0354] "Real-time" means that processing occurs immediately with little or no delay.

[0355] "Updating" means replacing data with new information.

[0356] This invention is a system consisting of a terminal and a server that transcribes the audio of videos watched by users in real time and displays them as subtitles, while also recognizing and displaying the user's emotional state.

[0357] System Configuration

[0358] Terminal

[0359] The device is a device that plays videos and extracts audio data. The scenario begins when a user selects a video using a video playback app on the device and starts playback. The device then captures audio data and performs emotion recognition. To do this, the device uses a MediaPlayer object to play the video's audio track and an AudioRecord object to capture audio data in real time. The device then uses a camera to capture the user's facial expressions and uses facial recognition technology to analyze their emotional state.

[0360] server

[0361] The server saves the received audio data in a temporary file and calls a speech recognition service to generate text data. The server then passes this audio data to a speech recognition service such as the Google Cloud Speech-to-Text API to generate corresponding text data. The server then adds a timestamp to the generated text data and converts it into subtitle format. The server then sends this text data to the device.

[0362] Hardware and Software

[0363] The device hardware includes a camera, microphone, and display. The software includes application software that runs on Android or iOS. The server side uses cloud services (such as Google Cloud or AWS®) for voice recognition and emotion analysis.

[0364] Data processing and calculation

[0365] The device captures audio data in real time. The audio data is divided into small chunks and sent sequentially to the server. The server converts the received audio data into text data, adds a timestamp, and formats it into subtitles. The device receives this subtitle data and displays it as an overlay on the video playback screen. At the same time, the device uses camera footage to analyze the user's facial expressions and update emotional data in real time.

[0366] Specific examples

[0367] For example, consider the case where a user is watching an educational video and the instructor says, "I will explain the basic syntax of Python." The device captures the audio and sends it to the server. The server converts the audio into text, generating the text "I will explain the basic syntax of Python." A timestamp is added to the text, and subtitle data is generated as follows:

[0368] 1

[0369] 00:00:01,000 --> 00:00:04,000

[0370] Learn about the basic syntax of Python

[0371] At the same time, the device captures the user's facial expressions with a camera and generates emotional data such as "excitement" or "concentration." This emotional data is also displayed with a timestamp, allowing the user to view subtitles containing emotional information in real time.

[0372] Prompt Sentence Examples

[0373] A real-time video subtitle generation and emotion recognition system. This application converts the audio of the video the user is watching into text, and simultaneously analyzes the user's emotions and adds them to the subtitles. Please write the following Python pseudocode:

[0374] Playing videos using OpenCV

[0375] Generate subtitles using the Google Cloud Speech-to-Text API

[0376] Recognizes and displays emotions using facial recognition and voice tone analysis

[0377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0378] Step 1:

[0379] The user launches a video playback application on their device, selects a video to watch, and starts playing it, which starts playing the video using a MediaPlayer object.

[0380] Input: Video file

[0381] Output: Playing video

[0382] Step 2:

[0383] The device captures the audio track of the video in real time by using an AudioRecord object to obtain audio data in real time.

[0384] Input: Video currently playing

[0385] Output: Real-time captured audio data

[0386] Step 3:

[0387] The device divides the captured audio data into chunks of a fixed size and sends them to the server sequentially. The transmission is performed asynchronously, for example, sending chunk data every second.

[0388] Input: Captured audio data

[0389] Output: Audio data chunks sent to the server

[0390] Step 4:

[0391] The server saves the received audio data chunks as temporary files and calls a speech recognition service (such as the Google Cloud Speech-to-Text API) to convert the audio data into text.

[0392] Input: Audio data chunks

[0393] Output: Text data

[0394] Step 5:

[0395] The server adds a timestamp to the generated text data and converts it into a subtitle format, for example, generating SRT format subtitle data.

[0396] Input: Text data

[0397] Output: Subtitle data with timestamps

[0398] Step 6:

[0399] The server transmits the converted subtitle data to the terminal.

[0400] Input: Timestamped subtitle data

[0401] Output: Subtitle data sent to the device

[0402] Step 7:

[0403] The device overlays the received subtitle data on the video playback screen, and the subtitles are displayed at the appropriate time based on the timestamp.

[0404] Input: Timestamped subtitle data

[0405] Output: Overlaid subtitles

[0406] Step 8:

[0407] The device uses a camera to capture the user's facial expressions, analyzes the facial expression data, and performs real-time emotion recognition using an emotion recognition engine.

[0408] Input: Camera image

[0409] Output: Real-time analyzed emotion data

[0410] Step 9:

[0411] In order for the device to display the analyzed emotion data, the subtitle data and emotion data are integrated and displayed on the video playback screen.

[0412] Input: Analyzed emotion data, subtitle data

[0413] Output: Subtitles with emotional information

[0414] Step 10:

[0415] While watching a video, users can check the subtitles and emotional information generated in real time, providing a better viewing experience.

[0416] Input: subtitles with emotional information

[0417] Output: High-quality viewing experience

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0419] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0421] [Second embodiment]

[0422] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0423] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0424] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0426] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0428] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0429] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0430] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0433] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0434] The present invention uses a system that combines a terminal and a server to transcribe the audio in a video in real time and display it as subtitles. The specific operation of this system is described below.

[0435] System Overview

[0436] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[0437] Program structure and operation

[0438] User operations

[0439] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[0440] Terminal side processing

[0441] The device extracts the video's audio track in real time as soon as the video starts playing, using a MediaPlayer object to play the audio and an AudioRecord object to capture the audio data.

[0442] The extracted audio data is divided into chunks of a predetermined size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[0443] Server-side processing

[0444] The server saves the received audio data in a temporary file and sends it to a speech recognition service such as the Google Cloud Speech-to-Text API. The text data generated by the speech recognition is returned to the server with a timestamp.

[0445] The server parses the returned text data and formats it into a subtitle format, usually .srt or WebVTT, and sends the formatted subtitle data to the device.

[0446] Subtitle display on the device

[0447] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[0448] Specific examples

[0449] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[0450] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[0451] 1

[0452] 00:00:01,000 --> 00:00:04,000

[0453] Hello, this is an introductory text for Python.

[0454] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[0455] Advantages of the invention

[0456] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[0457] The processing flow will be explained below.

[0458] Step 1:

[0459] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[0460] Step 2:

[0461] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[0462] Step 3:

[0463] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[0464] Step 4:

[0465] The server saves the received voice data as a temporary file, which is then ready to be sent to the voice recognition server.

[0466] Step 5:

[0467] The server calls a speech recognition service, such as the Google Cloud Speech-to-Text API, and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[0468] Step 6:

[0469] The server analyzes the received text data and timestamp information and formats it into .srt or WebVTT format. Specifically, it defines the text and its start and end times.

[0470] Step 7:

[0471] The server sends the formatted subtitle data to the terminal as an HTTP response.

[0472] Step 8:

[0473] The device analyzes the subtitle data received and processes it to display it in a TextView object. The subtitle display is implemented using Handlers and Runnables so that it is updated in real time based on timestamps.

[0474] Step 9:

[0475] When a user watches a video, the device overlays subtitles onto the video in real time, allowing the user to understand the video content not only audibly but also visually.

[0476] Example 1

[0477] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0478] Video content is consumed daily, but people with hearing impairments and those in noisy environments often find it difficult to understand the audio. Furthermore, when watching videos in foreign languages, there is a demand for a function that displays audio as text data. However, existing systems lack real-time capabilities and are limited in their use due to limitations on the devices and services they can be used with, resulting in a lack of flexibility. Furthermore, delays in the display of subtitles can degrade the user experience.

[0479] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0480] In this invention, the server includes means for performing speech recognition processing and generating corresponding character data, means for formatting the generated character data into a subtitle format and attaching a timestamp, and means for transmitting the formatted subtitle data to the terminal, thereby enabling accurate subtitle display in real time and improving the user experience.

[0481] A "terminal" is a device that plays video, extracts audio, and sends it to a server.

[0482] "Audio signal" refers to data captured in real time from audio data within a video.

[0483] "Real time" means that processing, from audio extraction to subtitle display, is carried out almost simultaneously.

[0484] The "predetermined size" is the unit by which the audio signal is divided into a specified interval (for example, every second).

[0485] "Asynchronous" means that audio data is transmitted in real time, but each transmission is carried out independently.

[0486] The "server" is a device that receives an audio signal, performs voice recognition processing, generates subtitle data, and transmits it to a terminal.

[0487] "Speech recognition processing" is a technology that converts voice signals into text data.

[0488] "Character data" is text-format data generated by voice recognition.

[0489] A "subtitle format" is a specific format (e.g., .srt or WebVTT) used to display audio in a video as text.

[0490] A "timestamp" is information that is added to character data and indicates the exact timing of display.

[0491] "Overlay display" is a method of displaying content over the video playback screen.

[0492] The present invention uses a system that combines a terminal and a server to transcribe audio from a video in real time and display the transcribed audio as subtitles. An embodiment of this system will be described in detail below.

[0493] System Overview

[0494] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[0495] User operations

[0496] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[0497] Terminal side processing

[0498] The device extracts the audio signal of the video in real time as soon as the video starts playing. To do this, it uses a MediaPlayer object to play the audio and an AudioRecord object to capture the audio signal. The captured audio signal is divided into chunks of a specified size and sent to the server asynchronously. Data is sent to the server every second.

[0499] Server-side processing

[0500] The server saves the received audio signal to a temporary file and sends the data to a speech recognition service (e.g., Google Cloud Speech-to-Text API). The text data generated by speech recognition is returned to the server with a timestamp. The server parses the returned text data and formats it into a subtitle format (e.g., .srt, WebVTT). The formatted subtitle data is then sent to the device.

[0501] Subtitle display on the device

[0502] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, allowing the user to view the subtitles in real time.

[0503] Specific examples

[0504] For example, consider the case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio signal and generates text data such as "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file as follows:

[0505] 1

[0506] 00:00:01,000 --> 00:00:04,000

[0507] Hello, this is an introductory text for Python.

[0508] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[0509] Prompt Sentence Examples

[0510] Prompt for explaining a program that uses generative AI models to generate real-time subtitles:

[0511] "Please tell me the process flow of a program that captures the audio of a video being played on the YouTube app and generates subtitles in real time using the Google Cloud Speech-to-Text API."

[0512] Advantages of the invention

[0513] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[0514] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0515] Step 1:

[0516] The user plays a video on their device. The input is the user selecting the video they want to play, and the output is the video starting to play on the device. Specifically, the user selects a video using the YouTube app or similar and presses the play button.

[0517] Step 2:

[0518] When the device starts playing the video, it also starts playing the audio signal using a MediaPlayer object. The input is the video file, and the output is the audio signal played in real time. Specifically, the MediaPlayer processes the audio signal and plays the audio.

[0519] Step 3:

[0520] The device uses an AudioRecord object to capture audio signals in real time. The input is the audio signal played by the MediaPlayer, and the output is the captured audio signal. Specifically, AudioRecord captures the audio signal and stores it in memory.

[0521] Step 4:

[0522] The device divides the captured audio signal into chunks of a specified size and sends them asynchronously to the server. The input is the captured audio signal, and the output is the audio signal data divided into chunks of a specified size (e.g., every second). Specifically, the audio signal is divided into chunks every second and sent to the server over the network.

[0523] Step 5:

[0524] The server saves the received audio signal in a temporary file. The input is the audio signal data sent from the terminal, and the output is the audio data saved in the temporary file. Specifically, the server saves the received audio signal as temp.wav.

[0525] Step 6:

[0526] The server sends the audio signal stored in a temporary file to the speech recognition service. The input is the temporary file of the stored audio signal, and the output is a request to the speech recognition service. Specifically, the server sends the audio file to the Google Cloud Speech-to-Text API and requests text data.

[0527] Step 7:

[0528] The speech recognition service analyzes the audio signal and generates corresponding text data. The input is the transmitted audio file, and the output is the generated text data. Specifically, the speech recognition service analyzes the audio signal and returns text data such as "Hello, this is an introductory text for Python."

[0529] Step 8:

[0530] The server adds a timestamp to the generated text data and formats it into subtitle format. The input is the text data returned from the speech recognition service, and the output is subtitle format data with a timestamp. Specifically, the text data is formatted into the following .srt format:

[0531] 1

[0532] 00:00:01,000 --> 00:00:04,000

[0533] Hello, this is an introductory text for Python.

[0534] Step 9:

[0535] The server sends the formatted subtitle data to the terminal. The input is time-stamped subtitle data, and the output is an API response to the terminal. Specifically, the server sends the formatted subtitle data to the terminal as an HTTP response.

[0536] Step 10:

[0537] The device analyzes the subtitle data received from the server and displays it as an overlay on the video playback screen. The input is the subtitle data sent from the server, and the output is real-time subtitles displayed on the video screen. Specifically, the device parses the received .srt file and displays the subtitles based on the timestamp. The user can view the subtitles in real time.

[0538] (Application example 1)

[0539] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0540] Current video streaming services do not adequately provide the functionality to display subtitles of audio content in real time while watching. This makes it difficult for the hearing impaired and users in multilingual environments to understand video content. In addition, the functionality available varies depending on the device, making it difficult to provide a consistent viewing experience for all users.

[0541] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0542] In this invention, the server includes means for performing speech recognition using received voice data and generating corresponding text data, means for assigning a timestamp to the generated text data, and means for converting the time-stamped text data into a subtitle format, thereby enabling subtitles to be generated and displayed in real time on a variety of playback devices.

[0543] A "terminal" is a device that extracts audio data, communicates with a server to receive text data, and displays the text data in a format that corresponds to the video data.

[0544] "Video data" refers to data of video content that a user views.

[0545] "Audio data" refers to data that indicates the audio portion included in the video data.

[0546] A "server" is a computer system that receives voice data sent from a terminal and performs voice recognition and generates text data.

[0547] "Speech recognition" is a process of converting received voice data into text data.

[0548] "Text data" is character string information generated by speech recognition.

[0549] A "timestamp" is time information added to text data, and indicates the timing of displaying subtitles.

[0550] The "subtitle format" is a data format for appropriately formatting the generated text data and displaying it in correspondence with the video data.

[0551] A "video distribution service" is an online service that distributes video content over the Internet.

[0552] "Real-time subtitle generation" is a function that instantly converts audio data into text data and displays it while a video is playing.

[0553] "A wide variety of playback devices" refers to various types of devices, such as smartphones, smart glasses, and head-mounted displays.

[0554] The present invention provides a system for generating and displaying subtitles in real time for video distribution services. This system is configured by combining a terminal, a server, and a voice recognition service.

[0555] System Configuration

[0556] 1. Device:

[0557] The hardware used includes smartphones, smart glasses, and head-mounted displays.

[0558] Audio data is extracted from the video data simultaneously with the video playback.

[0559] To capture audio data, on a smartphone, MediaPlayer (Android) or AVPlayer (iOS) is used, and AudioRecord (Android) or AVAudioRecorder (iOS) is used to record the audio data.

[0560] 2. Server:

[0561] Receives voice data sent from the terminal and performs voice recognition.

[0562] The Google Cloud Speech-to-Text API is used as the voice recognition service.

[0563] The text data generated by speech recognition is time-stamped and formatted into .srt or WebVTT format.

[0564] 3. Data flow:

[0565] When the device starts playing the video, it extracts the audio data and divides it into chunks of a specified size (e.g., 1 second).

[0566] The device asynchronously sends this to the server, which temporarily stores the received audio data and sends it to the Google Cloud Speech-to-Text API.

[0567] It receives the text data returned as a result of speech recognition, adds a timestamp to it, and formats it into subtitle format.

[0568] The formatted subtitle data is sent to the terminal, which receives it and displays it as subtitles.

[0569] Specific examples

[0570] Consider a specific example where a user is watching a movie on a smartphone using a video streaming service. When the user starts playing the video, the device captures the audio data and sends it to a server. The server then sends the received audio data to a speech recognition service, which generates text data such as "Once upon a time...". The generated text data is then given a timestamp and formatted in .srt format. This subtitle data is then sent to the device and displayed on the video screen in real time.

[0571] Prompt Sentence Examples

[0572] Develop an application that captures the audio track while a user is watching a movie on a video streaming service on their smartphone, converts it into text using the Google Cloud Speech-to-Text API, and displays it as real-time subtitles.

[0573] This system makes it possible to generate and display subtitles in real time on a variety of devices for video streaming services, thereby providing a consistent, high-quality viewing experience for users with hearing impairments and those in multilingual environments.

[0574] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0575] Step 1:

[0576] A user launches a video streaming service application on a device (smartphone, smart glasses, head-mounted display) and starts playing a video. At this point, the audio data extraction process begins along with the playback of the video data. Specifically, the device uses MediaPlayer (Android) or AVPlayer (iOS) to play the audio track of the video data in real time, and simultaneously captures the audio data using AudioRecord (Android) or AVAudioRecorder (iOS).

[0577] Input: Video playback start command, video data

[0578] Output: Extracted audio data

[0579] Step 2:

[0580] The extracted audio data is divided into chunks of a specified size (e.g., 1 second). The device sends this audio data to the server asynchronously. The division and transmission of the audio data is performed continuously, and continues without interruption while the user is watching the video. This maintains real-time performance.

[0581] Input: Extracted audio data

[0582] Output: Audio data chunks divided into predetermined sizes

[0583] Step 3:

[0584] The server temporarily stores the received audio data chunks and sends them to a speech recognition service such as the Google Cloud Speech-to-Text API, which converts the audio data into text data.

[0585] Input: Audio data chunks divided into predetermined sizes

[0586] Output: Generated text data

[0587] Step 4:

[0588] The server appends a timestamp to the text data returned by the speech recognition service and formats it into a subtitle format such as .srt or WebVTT. During this formatting process, the text data and the timestamp are formatted accordingly.

[0589] Input: Generated text data

[0590] Output: Subtitle data with timestamps

[0591] Step 5:

[0592] The server sends the formatted subtitle data to the device. The device receives this data and displays it as an overlay on the video playback screen. The display timing is based on the timestamp in real time, allowing the user to visually understand the audio content.

[0593] Input: Formatted subtitle data

[0594] Output: Display subtitles on the video playback screen

[0595] Through these processing steps, the video streaming service application can transcribe the audio of the video being watched by the user in real time and display it as subtitles, making it easier for users with hearing impairments and those in multilingual environments to understand the video content.

[0596] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0597] This invention combines a system that transcribes the audio of video content in real time and displays it as subtitles with an emotion engine, recognizing the user's emotional state and displaying it together with the subtitle data, thereby providing video viewers with richer information.

[0598] System Overview

[0599] This system consists of a terminal and a server. The terminal is operated by the user and is responsible for playing videos, extracting audio data, and recognizing emotions. The server, on the other hand, is responsible for voice recognition and generating subtitles. Furthermore, an emotion engine recognizes the user's emotions and displays the results along with the subtitle data.

[0600] Program structure and operation

[0601] User operations

[0602] The user selects a video using a video playback app on their device and starts playing it, while the system works in the background, extracting audio and recognizing emotions.

[0603] Terminal side processing

[0604] When the device starts playing the video, it uses the MediaPlayer object to play the video's audio track, and simultaneously captures the audio data in real time using the AudioRecord object. The captured audio data is divided into chunks of a specified size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[0605] The emotion engine also analyzes the user's facial expressions and tone of voice using the user's camera footage and audio signals. The analyzed emotion data is updated in real time.

[0606] Server-side processing

[0607] The server saves the received audio data to a temporary file, then calls a speech recognition service such as the Google Cloud Speech-to-Text API and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[0608] The server parses the returned text data and formats it into subtitles, which are then sent to the device.

[0609] Subtitle display on the device

[0610] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp. The analyzed emotional data is also displayed at the same time. This allows the user to understand not only the audio information but also the emotional state.

[0611] Specific examples

[0612] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[0613] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[0614] 1

[0615] 00:00:01,000 --> 00:00:04,000

[0616] Hello, this is an introductory text for Python.

[0617] At the same time, the emotion engine analyzes the user's facial expressions and tone of voice to generate emotion data such as "excitement" or "concentration." For example, the following display is generated:

[0618] 1

[0619] 00:00:01,000 --> 00:00:04,000

[0620] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[0621] The device receives this subtitle data and emotion data and displays them on the video playback screen, allowing the user to view the emotion information along with the subtitles in real time.

[0622] Advantages of the invention

[0623] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, analyzing and displaying the user's emotional information can further enrich the viewing experience. Another major advantage is the simple system configuration, which combines ease of implementation with high accuracy by utilizing existing cloud services.

[0624] The processing flow will be explained below.

[0625] Step 1:

[0626] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[0627] Step 2:

[0628] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[0629] Step 3:

[0630] To perform real-time emotion recognition, the device captures the user's face with a camera and the user's voice tone with a microphone, and these data are analyzed by the emotion engine.

[0631] Step 4:

[0632] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[0633] Step 5:

[0634] The server saves the received voice data as a temporary file and sends it to a speech recognition service such as Google Cloud Speech-to-Text API, which converts the voice data into text data and returns a response with a timestamp.

[0635] Step 6:

[0636] The server analyzes the received text data and its timestamp information and formats it into subtitle format (for example, .srt format).

[0637] Step 7:

[0638] The server sends the formatted subtitle data to the terminal as an HTTP response.

[0639] Step 8:

[0640] The device analyzes the received text data and processes it to overlay it on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[0641] Step 9:

[0642] The device displays the analyzed emotion data along with the text data, which includes the user's emotional state (e.g., sad, happy, etc.) analyzed by the emotion engine.

[0643] Step 10:

[0644] Users can view their own emotional state along with subtitles while watching videos in real time, improving the viewing experience and providing emotional feedback.

[0645] Example 2

[0646] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] Conventional subtitling systems primarily convert audio data into text data for display, and are unable to reflect the user's emotional information. Furthermore, they lack the ingenuity to further enrich the user's viewing experience, limiting the information they can provide to viewers. As a result, there is a need for a method to further deepen the understanding of video content, especially for the hearing impaired and in environments requiring multilingual support.

[0648] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for extracting audio data from video data at the terminal; means for transmitting the extracted audio data to the server; means for performing speech recognition using the audio data received at the server and generating corresponding text data; means for transmitting the generated text data to the terminal; means for displaying the received text data at the terminal in a form corresponding to the video data; means for analyzing the user's facial expression and tone of voice at the terminal and generating emotional data of the user; and means for displaying the generated emotional data together with the text data in a form corresponding to the video data. This further enriches the user's viewing experience and enables a deeper understanding of video content, particularly for the hearing impaired or in an environment requiring multilingual support.

[0649] A "terminal" is a device that is operated by a user and has the functions of playing back video data, extracting audio data, and generating emotion data.

[0650] "Video data" refers to data that includes visual information of moving image content, and is played back in synchronization with audio data.

[0651] "Audio data" refers to the audio included in the video data, and is sound information that can be heard by the user.

[0652] A "server" is a computing device that receives data sent from a terminal and performs voice recognition and data conversion.

[0653] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[0654] "Text data" refers to data containing character information obtained by voice recognition, and is displayed in correspondence with video data.

[0655] "Emotion data" is data that indicates the emotional state of the user, obtained by analyzing the user's facial expressions and tone of voice.

[0656] The "subtitle format" is a format in which a timestamp is added to text data and displayed in synchronization with video data.

[0657] "Platform" means a system that includes the underlying software and hardware for performing video playback, audio capture, and sentiment analysis.

[0658] This system transcribes the audio of video content in real time, displays it as subtitles, and simultaneously recognizes the user's emotional state and displays it together with the subtitle data. This system consists of a terminal operated by the user and a server that processes the data.

[0659] A device is a device on which a user selects a video using a video playback app and starts playback. The device is equipped with a MediaPlayer object, which plays the video's audio track, and simultaneously captures audio data in real time using an AudioRecord object. The captured audio data is stored in a buffer and then split into 1-second chunks and sent asynchronously to the server.

[0660] The device also comes equipped with an emotion engine that uses camera footage and microphone input to analyze facial expressions and vocal tone, generating emotion data that is updated in real time.

[0661] The server receives the audio data sent from the device and saves it as a temporary file. This saved audio data is then converted into text data using a speech recognition service such as the Google Cloud Speech-to-Text API. The text data is given a timestamp and is analyzed and formatted within the server. The formatted subtitle data is then sent back to the device.

[0662] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, and the analyzed emotional data is also displayed. This allows the user to understand not only the audio information but also the emotional state of the person.

[0663] For example, if a user plays a video titled "Introduction to Python," the device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio data and generates the text "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file and displayed as follows:

[0664] srt

[0665] 1

[0666] 00:00:01,000 --> 00:00:04,000

[0667] Hello, this is an introductory text for Python.

[0668] At the same time, the emotion engine analyzes the user's facial expressions and voice tone to generate emotion data such as "excitement" or "concentration." The device receives this subtitle data and emotion data and displays it in real time on the video playback screen. For example, it can be displayed as follows:

[0669] srt

[0670] 1

[0671] 00:00:01,000 --> 00:00:04,000

[0672] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[0673] An example of a prompt is, "Please explain the detailed process flow of a system that converts video audio into text in real time and displays it as subtitles, while also analyzing user emotions."

[0674] This invention allows users to enjoy subtitles in real time, deepening their understanding of video content, especially for the hearing impaired and in environments where multilingual support is required. Another major feature is that it can enrich the viewing experience by analyzing and displaying emotional information.

[0675] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0676] Step 1:

[0677] User plays a video

[0678] A user uses a video playback app to select a video and begin playing it.

[0679] Input: Select a video file and trigger playback.

[0680] Output: Video playback start signal.

[0681] Step 2:

[0682] Capture audio data

[0683] The device plays the audio track of the selected video using a MediaPlayer object, while simultaneously capturing the audio data in real time using an AudioRecord object.

[0684] Input: The audio track of the video file.

[0685] Output: Real-time captured audio data.

[0686] Specific operation: The captured audio data is temporarily stored in a buffer and divided into 1-second chunks.

[0687] Step 3:

[0688] Capturing Emotional Data

[0689] The device uses the user's camera footage and microphone input to analyze the user's facial expressions and voice tone using an emotion engine.

[0690] Input: User's camera video and audio input.

[0691] Output: Real-time analyzed emotion data.

[0692] Specific operation: By recognizing facial expressions from camera footage and analyzing the tone of voice from microphone input, an emotional state (such as "excitement" or "concentration") is generated.

[0693] Step 4:

[0694] Sending voice data to the server

[0695] The device asynchronously sends captured audio data to the server every second.

[0696] Input: Segmented audio data.

[0697] Output: Audio data sent to server completed.

[0698] Specific operation: The divided audio data is sent to the server using a protocol such as an HTTP request.

[0699] Step 5:

[0700] Saving audio data

[0701] The server saves the received audio data as a temporary file.

[0702] Input: Transmitted audio data.

[0703] Output: Audio data saved to a temporary file.

[0704] Specific operation: The audio data is saved as a file in WAV format or similar, and a timestamp is added to the file name.

[0705] Step 6:

[0706] Transcription of audio data

[0707] The server sends the stored audio data to the Google Cloud Speech-to-Text API and converts it into text data.

[0708] Input: Saved audio data file.

[0709] Output: Text data (with timestamp).

[0710] Specific operation: Calls the speech recognition service and converts the speech data into text format.

[0711] Step 7:

[0712] Subtitle data formatting

[0713] The server analyzes the converted text data and formats it into time-stamped subtitle data (such as .srt format).

[0714] Input: Text data generated by speech recognition.

[0715] Output: Formatted subtitle data.

[0716] Specific operation: Add appropriate line breaks and timestamps to the text data and format it into subtitle format.

[0717] Step 8:

[0718] Subtitle data transmission to device

[0719] The server sends the formatted subtitle data to the terminal.

[0720] Input: Formatted subtitle data.

[0721] Output: Subtitle data has been sent to the device.

[0722] Specific operation: Subtitle data is sent to the device using a protocol such as an HTTP request.

[0723] Step 9:

[0724] Displaying subtitle data and emotion data

[0725] The device analyzes the subtitle data received and the generated emotional data, and overlays it on the video playback screen in real time.

[0726] Input: Received subtitle data, parsed emotion data.

[0727] Output: Subtitles and emotions on the video playback screen.

[0728] Specific operation: Subtitle data and emotion data are synchronized based on timestamps and displayed in the appropriate position.

[0729] (Application example 2)

[0730] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] Conventional video playback systems have difficulty displaying subtitles in real time while a user is watching a video. Furthermore, they lack emotion recognition capabilities and are unable to display subtitles that take the user's emotional state into account, limiting the viewing experience. Real-time subtitle generation and emotional information display are particularly necessary for environments requiring support for the hearing impaired and multilingual support.

[0732] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for performing speech recognition and generating corresponding text data, means for assigning a timestamp to the generated text data and converting it into a subtitle format, and means for displaying emotion data generated in real time together with subtitles based on the timestamp. This makes it possible to generate subtitles in real time and display subtitles according to the user's emotional state.

[0733] A "terminal" is a device operated by a user that plays videos, extracts audio data, and recognizes emotions.

[0734] "Video data" refers to visual information contained in video content.

[0735] "Audio data" refers to the data of the audio track included in the video.

[0736] "Extraction" is the operation of extracting specific information (in this case, audio data) from a video.

[0737] "Transmit" means transferring data from one device to another device or server.

[0738] A "server" is a computer system that performs processes such as speech recognition and text generation.

[0739] "Speech recognition" is a technology that analyzes voice data and generates corresponding text data.

[0740] "Text data" is character information generated by voice recognition.

[0741] "Subtitle data" is text data to which a timestamp is added, and is displayed superimposed on video data.

[0742] A "timestamp" is time information that indicates a specific moment.

[0743] A "subtitle format" is a particular format in which text is displayed in video data.

[0744] "Facial expressions" are movements that show emotions and reactions on the user's face.

[0745] "Identification" is the act of analyzing video data and facial expression data to extract specific features and information.

[0746] An "emotional state" is information that describes the emotions a user is experiencing at a particular moment.

[0747] "Analysis" refers to examining data in detail to clarify its structure and meaning.

[0748] "Real-time" means that processing occurs immediately with little or no delay.

[0749] "Updating" means replacing data with new information.

[0750] This invention is a system consisting of a terminal and a server that transcribes the audio of videos watched by users in real time and displays them as subtitles, while also recognizing and displaying the user's emotional state.

[0751] System Configuration

[0752] Terminal

[0753] The device is a device that plays videos and extracts audio data. The scenario begins when a user selects a video using a video playback app on the device and starts playback. The device then captures audio data and performs emotion recognition. To do this, the device uses a MediaPlayer object to play the video's audio track and an AudioRecord object to capture audio data in real time. The device then uses a camera to capture the user's facial expressions and uses facial recognition technology to analyze their emotional state.

[0754] server

[0755] The server saves the received audio data in a temporary file and calls a speech recognition service to generate text data. The server then passes this audio data to a speech recognition service such as the Google Cloud Speech-to-Text API to generate corresponding text data. The server then adds a timestamp to the generated text data and converts it into subtitle format. The server then sends this text data to the device.

[0756] Hardware and Software

[0757] The device hardware includes a camera, microphone, and display. The software includes application software that runs on Android or iOS. On the server side, cloud services (such as Google Cloud or AWS) are used for voice recognition and sentiment analysis.

[0758] Data processing and calculation

[0759] The device captures audio data in real time. The audio data is divided into small chunks and sent sequentially to the server. The server converts the received audio data into text data, adds a timestamp, and formats it into subtitles. The device receives this subtitle data and displays it as an overlay on the video playback screen. At the same time, the device uses camera footage to analyze the user's facial expressions and update emotional data in real time.

[0760] Specific examples

[0761] For example, consider the case where a user is watching an educational video and the instructor says, "I will explain the basic syntax of Python." The device captures the audio and sends it to the server. The server converts the audio into text, generating the text "I will explain the basic syntax of Python." A timestamp is added to the text, and subtitle data is generated as follows:

[0762] 1

[0763] 00:00:01,000 --> 00:00:04,000

[0764] Learn about the basic syntax of Python

[0765] At the same time, the device captures the user's facial expressions with a camera and generates emotional data such as "excitement" or "concentration." This emotional data is also displayed with a timestamp, allowing the user to view subtitles containing emotional information in real time.

[0766] Prompt Sentence Examples

[0767] A real-time video subtitle generation and emotion recognition system. This application converts the audio of the video the user is watching into text, and simultaneously analyzes the user's emotions and adds them to the subtitles. Please write the following Python pseudocode:

[0768] Playing videos using OpenCV

[0769] Generate subtitles using the Google Cloud Speech-to-Text API

[0770] Recognizes and displays emotions using facial recognition and voice tone analysis

[0771] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0772] Step 1:

[0773] The user launches a video playback application on their device, selects a video to watch, and starts playing it, which starts playing the video using a MediaPlayer object.

[0774] Input: Video file

[0775] Output: Playing video

[0776] Step 2:

[0777] The device captures the audio track of the video in real time by using an AudioRecord object to obtain audio data in real time.

[0778] Input: Video currently playing

[0779] Output: Real-time captured audio data

[0780] Step 3:

[0781] The device divides the captured audio data into chunks of a fixed size and sends them to the server sequentially. The transmission is performed asynchronously, for example, sending chunk data every second.

[0782] Input: Captured audio data

[0783] Output: Audio data chunks sent to the server

[0784] Step 4:

[0785] The server saves the received audio data chunks as temporary files and calls a speech recognition service (such as the Google Cloud Speech-to-Text API) to convert the audio data into text.

[0786] Input: Audio data chunks

[0787] Output: Text data

[0788] Step 5:

[0789] The server adds a timestamp to the generated text data and converts it into a subtitle format, for example, generating SRT format subtitle data.

[0790] Input: Text data

[0791] Output: Subtitle data with timestamps

[0792] Step 6:

[0793] The server transmits the converted subtitle data to the terminal.

[0794] Input: Timestamped subtitle data

[0795] Output: Subtitle data sent to the device

[0796] Step 7:

[0797] The device overlays the received subtitle data on the video playback screen, and the subtitles are displayed at the appropriate time based on the timestamp.

[0798] Input: Timestamped subtitle data

[0799] Output: Overlaid subtitles

[0800] Step 8:

[0801] The device uses a camera to capture the user's facial expressions, analyzes the facial expression data, and performs real-time emotion recognition using an emotion recognition engine.

[0802] Input: Camera image

[0803] Output: Real-time analyzed emotion data

[0804] Step 9:

[0805] In order for the device to display the analyzed emotion data, the subtitle data and emotion data are integrated and displayed on the video playback screen.

[0806] Input: Analyzed emotion data, subtitle data

[0807] Output: Subtitles with emotional information

[0808] Step 10:

[0809] While watching a video, users can check the subtitles and emotional information generated in real time, providing a better viewing experience.

[0810] Input: subtitles with emotional information

[0811] Output: High-quality viewing experience

[0812] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0813] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0814] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0815] [Third embodiment]

[0816] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0817] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0818] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0819] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0820] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0821] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0822] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0823] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0824] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0825] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0826] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0827] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0828] The present invention uses a system that combines a terminal and a server to transcribe the audio in a video in real time and display it as subtitles. The specific operation of this system is described below.

[0829] System Overview

[0830] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[0831] Program structure and operation

[0832] User operations

[0833] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[0834] Terminal side processing

[0835] The device extracts the video's audio track in real time as soon as the video starts playing, using a MediaPlayer object to play the audio and an AudioRecord object to capture the audio data.

[0836] The extracted audio data is divided into chunks of a predetermined size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[0837] Server-side processing

[0838] The server saves the received audio data in a temporary file and sends it to a speech recognition service such as the Google Cloud Speech-to-Text API. The text data generated by the speech recognition is returned to the server with a timestamp.

[0839] The server parses the returned text data and formats it into a subtitle format, usually .srt or WebVTT, and sends the formatted subtitle data to the device.

[0840] Subtitle display on the device

[0841] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[0842] Specific examples

[0843] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[0844] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[0845] 1

[0846] 00:00:01,000 --> 00:00:04,000

[0847] Hello, this is an introductory text for Python.

[0848] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[0849] Advantages of the invention

[0850] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[0851] The processing flow will be explained below.

[0852] Step 1:

[0853] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[0854] Step 2:

[0855] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[0856] Step 3:

[0857] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[0858] Step 4:

[0859] The server saves the received voice data as a temporary file, which is then ready to be sent to the voice recognition server.

[0860] Step 5:

[0861] The server calls a speech recognition service, such as the Google Cloud Speech-to-Text API, and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[0862] Step 6:

[0863] The server analyzes the received text data and timestamp information and formats it into .srt or WebVTT format. Specifically, it defines the text and its start and end times.

[0864] Step 7:

[0865] The server sends the formatted subtitle data to the terminal as an HTTP response.

[0866] Step 8:

[0867] The device analyzes the subtitle data received and processes it to display it in a TextView object. The subtitle display is implemented using Handlers and Runnables so that it is updated in real time based on timestamps.

[0868] Step 9:

[0869] When a user watches a video, the device overlays subtitles onto the video in real time, allowing the user to understand the video content not only audibly but also visually.

[0870] Example 1

[0871] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0872] Video content is consumed daily, but people with hearing impairments and those in noisy environments often find it difficult to understand the audio. Furthermore, when watching videos in foreign languages, there is a demand for a function that displays audio as text data. However, existing systems lack real-time capabilities and are limited in their use due to limitations on the devices and services they can be used with, resulting in a lack of flexibility. Furthermore, delays in the display of subtitles can degrade the user experience.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0874] In this invention, the server includes means for performing speech recognition processing and generating corresponding character data, means for formatting the generated character data into a subtitle format and attaching a timestamp, and means for transmitting the formatted subtitle data to the terminal, thereby enabling accurate subtitle display in real time and improving the user experience.

[0875] A "terminal" is a device that plays video, extracts audio, and sends it to a server.

[0876] "Audio signal" refers to data captured in real time from audio data within a video.

[0877] "Real time" means that processing, from audio extraction to subtitle display, is carried out almost simultaneously.

[0878] The "predetermined size" is the unit by which the audio signal is divided into a specified interval (for example, every second).

[0879] "Asynchronous" means that audio data is transmitted in real time, but each transmission is carried out independently.

[0880] The "server" is a device that receives an audio signal, performs voice recognition processing, generates subtitle data, and transmits it to a terminal.

[0881] "Speech recognition processing" is a technology that converts voice signals into text data.

[0882] "Character data" is text-format data generated by voice recognition.

[0883] A "subtitle format" is a specific format (e.g., .srt or WebVTT) used to display audio in a video as text.

[0884] A "timestamp" is information that is added to character data and indicates the exact timing of display.

[0885] "Overlay display" is a method of displaying content over the video playback screen.

[0886] The present invention uses a system that combines a terminal and a server to transcribe audio from a video in real time and display the transcribed audio as subtitles. An embodiment of this system will be described in detail below.

[0887] System Overview

[0888] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[0889] User operations

[0890] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[0891] Terminal side processing

[0892] The device extracts the audio signal of the video in real time as soon as the video starts playing. To do this, it uses a MediaPlayer object to play the audio and an AudioRecord object to capture the audio signal. The captured audio signal is divided into chunks of a specified size and sent to the server asynchronously. Data is sent to the server every second.

[0893] Server-side processing

[0894] The server saves the received audio signal to a temporary file and sends the data to a speech recognition service (e.g., Google Cloud Speech-to-Text API). The text data generated by speech recognition is returned to the server with a timestamp. The server parses the returned text data and formats it into a subtitle format (e.g., .srt, WebVTT). The formatted subtitle data is then sent to the device.

[0895] Subtitle display on the device

[0896] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, allowing the user to view the subtitles in real time.

[0897] Specific examples

[0898] For example, consider the case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio signal and generates text data such as "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file as follows:

[0899] 1

[0900] 00:00:01,000 --> 00:00:04,000

[0901] Hello, this is an introductory text for Python.

[0902] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[0903] Prompt Sentence Examples

[0904] Prompt for explaining a program that uses generative AI models to generate real-time subtitles:

[0905] "Please tell me the process flow of a program that captures the audio of a video being played on the YouTube app and generates subtitles in real time using the Google Cloud Speech-to-Text API."

[0906] Advantages of the invention

[0907] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[0908] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0909] Step 1:

[0910] The user plays a video on their device. The input is the user selecting the video they want to play, and the output is the video starting to play on the device. Specifically, the user selects a video using the YouTube app or similar and presses the play button.

[0911] Step 2:

[0912] When the device starts playing the video, it also starts playing the audio signal using a MediaPlayer object. The input is the video file, and the output is the audio signal played in real time. Specifically, the MediaPlayer processes the audio signal and plays the audio.

[0913] Step 3:

[0914] The device uses an AudioRecord object to capture audio signals in real time. The input is the audio signal played by the MediaPlayer, and the output is the captured audio signal. Specifically, AudioRecord captures the audio signal and stores it in memory.

[0915] Step 4:

[0916] The device divides the captured audio signal into chunks of a specified size and sends them asynchronously to the server. The input is the captured audio signal, and the output is the audio signal data divided into chunks of a specified size (e.g., every second). Specifically, the audio signal is divided into chunks every second and sent to the server over the network.

[0917] Step 5:

[0918] The server saves the received audio signal in a temporary file. The input is the audio signal data sent from the terminal, and the output is the audio data saved in the temporary file. Specifically, the server saves the received audio signal as temp.wav.

[0919] Step 6:

[0920] The server sends the audio signal stored in a temporary file to the speech recognition service. The input is the temporary file of the stored audio signal, and the output is a request to the speech recognition service. Specifically, the server sends the audio file to the Google Cloud Speech-to-Text API and requests text data.

[0921] Step 7:

[0922] The speech recognition service analyzes the audio signal and generates corresponding text data. The input is the transmitted audio file, and the output is the generated text data. Specifically, the speech recognition service analyzes the audio signal and returns text data such as "Hello, this is an introductory text for Python."

[0923] Step 8:

[0924] The server adds a timestamp to the generated text data and formats it into subtitle format. The input is the text data returned from the speech recognition service, and the output is subtitle format data with a timestamp. Specifically, the text data is formatted into the following .srt format:

[0925] 1

[0926] 00:00:01,000 --> 00:00:04,000

[0927] Hello, this is an introductory text for Python.

[0928] Step 9:

[0929] The server sends the formatted subtitle data to the terminal. The input is time-stamped subtitle data, and the output is an API response to the terminal. Specifically, the server sends the formatted subtitle data to the terminal as an HTTP response.

[0930] Step 10:

[0931] The device analyzes the subtitle data received from the server and displays it as an overlay on the video playback screen. The input is the subtitle data sent from the server, and the output is real-time subtitles displayed on the video screen. Specifically, the device parses the received .srt file and displays the subtitles based on the timestamp. The user can view the subtitles in real time.

[0932] (Application example 1)

[0933] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0934] Current video streaming services do not adequately provide the functionality to display subtitles of audio content in real time while watching. This makes it difficult for the hearing impaired and users in multilingual environments to understand video content. In addition, the functionality available varies depending on the device, making it difficult to provide a consistent viewing experience for all users.

[0935] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0936] In this invention, the server includes means for performing speech recognition using received voice data and generating corresponding text data, means for assigning a timestamp to the generated text data, and means for converting the time-stamped text data into a subtitle format, thereby enabling subtitles to be generated and displayed in real time on a variety of playback devices.

[0937] A "terminal" is a device that extracts audio data, communicates with a server to receive text data, and displays the text data in a format that corresponds to the video data.

[0938] "Video data" refers to data of video content that a user views.

[0939] "Audio data" refers to data that indicates the audio portion included in the video data.

[0940] A "server" is a computer system that receives voice data sent from a terminal and performs voice recognition and generates text data.

[0941] "Speech recognition" is a process of converting received voice data into text data.

[0942] "Text data" is character string information generated by speech recognition.

[0943] A "timestamp" is time information added to text data, and indicates the timing of displaying subtitles.

[0944] The "subtitle format" is a data format for appropriately formatting the generated text data and displaying it in correspondence with the video data.

[0945] A "video distribution service" is an online service that distributes video content over the Internet.

[0946] "Real-time subtitle generation" is a function that instantly converts audio data into text data and displays it while a video is playing.

[0947] "A wide variety of playback devices" refers to various types of devices, such as smartphones, smart glasses, and head-mounted displays.

[0948] The present invention provides a system for generating and displaying subtitles in real time for video distribution services. This system is configured by combining a terminal, a server, and a voice recognition service.

[0949] System Configuration

[0950] 1. Device:

[0951] The hardware used includes smartphones, smart glasses, and head-mounted displays.

[0952] Audio data is extracted from the video data simultaneously with the video playback.

[0953] To capture audio data, on a smartphone, MediaPlayer (Android) or AVPlayer (iOS) is used, and AudioRecord (Android) or AVAudioRecorder (iOS) is used to record the audio data.

[0954] 2. Server:

[0955] Receives voice data sent from the terminal and performs voice recognition.

[0956] The Google Cloud Speech-to-Text API is used as the voice recognition service.

[0957] The text data generated by speech recognition is time-stamped and formatted into .srt or WebVTT format.

[0958] 3. Data flow:

[0959] When the device starts playing the video, it extracts the audio data and divides it into chunks of a specified size (e.g., 1 second).

[0960] The device asynchronously sends this to the server, which temporarily stores the received audio data and sends it to the Google Cloud Speech-to-Text API.

[0961] It receives the text data returned as a result of speech recognition, adds a timestamp to it, and formats it into subtitle format.

[0962] The formatted subtitle data is sent to the terminal, which receives it and displays it as subtitles.

[0963] Specific examples

[0964] Consider a specific example where a user is watching a movie on a smartphone using a video streaming service. When the user starts playing the video, the device captures the audio data and sends it to a server. The server then sends the received audio data to a speech recognition service, which generates text data such as "Once upon a time...". The generated text data is then given a timestamp and formatted in .srt format. This subtitle data is then sent to the device and displayed on the video screen in real time.

[0965] Prompt Sentence Examples

[0966] Develop an application that captures the audio track while a user is watching a movie on a video streaming service on their smartphone, converts it into text using the Google Cloud Speech-to-Text API, and displays it as real-time subtitles.

[0967] This system makes it possible to generate and display subtitles in real time on a variety of devices for video streaming services, thereby providing a consistent, high-quality viewing experience for users with hearing impairments and those in multilingual environments.

[0968] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0969] Step 1:

[0970] A user launches a video streaming service application on a device (smartphone, smart glasses, head-mounted display) and starts playing a video. At this point, the audio data extraction process begins along with the playback of the video data. Specifically, the device uses MediaPlayer (Android) or AVPlayer (iOS) to play the audio track of the video data in real time, and simultaneously captures the audio data using AudioRecord (Android) or AVAudioRecorder (iOS).

[0971] Input: Video playback start command, video data

[0972] Output: Extracted audio data

[0973] Step 2:

[0974] The extracted audio data is divided into chunks of a specified size (e.g., 1 second). The device sends this audio data to the server asynchronously. The division and transmission of the audio data is performed continuously, and continues without interruption while the user is watching the video. This maintains real-time performance.

[0975] Input: Extracted audio data

[0976] Output: Audio data chunks divided into predetermined sizes

[0977] Step 3:

[0978] The server temporarily stores the received audio data chunks and sends them to a speech recognition service such as the Google Cloud Speech-to-Text API, which converts the audio data into text data.

[0979] Input: Audio data chunks divided into predetermined sizes

[0980] Output: Generated text data

[0981] Step 4:

[0982] The server appends a timestamp to the text data returned by the speech recognition service and formats it into a subtitle format such as .srt or WebVTT. During this formatting process, the text data and the timestamp are formatted accordingly.

[0983] Input: Generated text data

[0984] Output: Subtitle data with timestamps

[0985] Step 5:

[0986] The server sends the formatted subtitle data to the device. The device receives this data and displays it as an overlay on the video playback screen. The display timing is based on the timestamp in real time, allowing the user to visually understand the audio content.

[0987] Input: Formatted subtitle data

[0988] Output: Display subtitles on the video playback screen

[0989] Through these processing steps, the video streaming service application can transcribe the audio of the video being watched by the user in real time and display it as subtitles, making it easier for users with hearing impairments and those in multilingual environments to understand the video content.

[0990] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0991] This invention combines a system that transcribes the audio of video content in real time and displays it as subtitles with an emotion engine, recognizing the user's emotional state and displaying it together with the subtitle data, thereby providing video viewers with richer information.

[0992] System Overview

[0993] This system consists of a terminal and a server. The terminal is operated by the user and is responsible for playing videos, extracting audio data, and recognizing emotions. The server, on the other hand, is responsible for voice recognition and generating subtitles. Furthermore, an emotion engine recognizes the user's emotions and displays the results along with the subtitle data.

[0994] Program structure and operation

[0995] User operations

[0996] The user selects a video using a video playback app on their device and starts playing it, while the system works in the background, extracting audio and recognizing emotions.

[0997] Terminal side processing

[0998] When the device starts playing the video, it uses the MediaPlayer object to play the video's audio track, and simultaneously captures the audio data in real time using the AudioRecord object. The captured audio data is divided into chunks of a specified size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[0999] The emotion engine also analyzes the user's facial expressions and tone of voice using the user's camera footage and audio signals. The analyzed emotion data is updated in real time.

[1000] Server-side processing

[1001] The server saves the received audio data to a temporary file, then calls a speech recognition service such as the Google Cloud Speech-to-Text API and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[1002] The server parses the returned text data and formats it into subtitles, which are then sent to the device.

[1003] Subtitle display on the device

[1004] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp. The analyzed emotional data is also displayed at the same time. This allows the user to understand not only the audio information but also the emotional state.

[1005] Specific examples

[1006] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[1007] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[1008] 1

[1009] 00:00:01,000 --> 00:00:04,000

[1010] Hello, this is an introductory text for Python.

[1011] At the same time, the emotion engine analyzes the user's facial expressions and tone of voice to generate emotion data such as "excitement" or "concentration." For example, the following display is generated:

[1012] 1

[1013] 00:00:01,000 --> 00:00:04,000

[1014] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[1015] The device receives this subtitle data and emotion data and displays them on the video playback screen, allowing the user to view the emotion information along with the subtitles in real time.

[1016] Advantages of the invention

[1017] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, analyzing and displaying the user's emotional information can further enrich the viewing experience. Another major advantage is the simple system configuration, which combines ease of implementation with high accuracy by utilizing existing cloud services.

[1018] The processing flow will be explained below.

[1019] Step 1:

[1020] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[1021] Step 2:

[1022] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[1023] Step 3:

[1024] To perform real-time emotion recognition, the device captures the user's face with a camera and the user's voice tone with a microphone, and these data are analyzed by the emotion engine.

[1025] Step 4:

[1026] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[1027] Step 5:

[1028] The server saves the received voice data as a temporary file and sends it to a speech recognition service such as Google Cloud Speech-to-Text API, which converts the voice data into text data and returns a response with a timestamp.

[1029] Step 6:

[1030] The server analyzes the received text data and its timestamp information and formats it into subtitle format (for example, .srt format).

[1031] Step 7:

[1032] The server sends the formatted subtitle data to the terminal as an HTTP response.

[1033] Step 8:

[1034] The device analyzes the received text data and processes it to overlay it on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[1035] Step 9:

[1036] The device displays the analyzed emotion data along with the text data, which includes the user's emotional state (e.g., sad, happy, etc.) analyzed by the emotion engine.

[1037] Step 10:

[1038] Users can view their own emotional state along with subtitles while watching videos in real time, improving the viewing experience and providing emotional feedback.

[1039] Example 2

[1040] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1041] Conventional subtitling systems primarily convert audio data into text data for display, and are unable to reflect the user's emotional information. Furthermore, they lack the ingenuity to further enrich the user's viewing experience, limiting the information they can provide to viewers. As a result, there is a need for a method to further deepen the understanding of video content, especially for the hearing impaired and in environments requiring multilingual support.

[1042] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for extracting audio data from video data at the terminal; means for transmitting the extracted audio data to the server; means for performing speech recognition using the audio data received at the server and generating corresponding text data; means for transmitting the generated text data to the terminal; means for displaying the received text data at the terminal in a form corresponding to the video data; means for analyzing the user's facial expression and tone of voice at the terminal and generating emotional data of the user; and means for displaying the generated emotional data together with the text data in a form corresponding to the video data. This further enriches the user's viewing experience and enables a deeper understanding of video content, particularly for the hearing impaired or in an environment requiring multilingual support.

[1043] A "terminal" is a device that is operated by a user and has the functions of playing back video data, extracting audio data, and generating emotion data.

[1044] "Video data" refers to data that includes visual information of moving image content, and is played back in synchronization with audio data.

[1045] "Audio data" refers to the audio included in the video data, and is sound information that can be heard by the user.

[1046] A "server" is a computing device that receives data sent from a terminal and performs voice recognition and data conversion.

[1047] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[1048] "Text data" refers to data containing character information obtained by voice recognition, and is displayed in correspondence with video data.

[1049] "Emotion data" is data that indicates the emotional state of the user, obtained by analyzing the user's facial expressions and tone of voice.

[1050] The "subtitle format" is a format in which a timestamp is added to text data and displayed in synchronization with video data.

[1051] "Platform" means a system that includes the underlying software and hardware for performing video playback, audio capture, and sentiment analysis.

[1052] This system transcribes the audio of video content in real time, displays it as subtitles, and simultaneously recognizes the user's emotional state and displays it together with the subtitle data. This system consists of a terminal operated by the user and a server that processes the data.

[1053] A device is a device on which a user selects a video using a video playback app and starts playback. The device is equipped with a MediaPlayer object, which plays the video's audio track, and simultaneously captures audio data in real time using an AudioRecord object. The captured audio data is stored in a buffer and then split into 1-second chunks and sent asynchronously to the server.

[1054] The device also comes equipped with an emotion engine that uses camera footage and microphone input to analyze facial expressions and vocal tone, generating emotion data that is updated in real time.

[1055] The server receives the audio data sent from the device and saves it as a temporary file. This saved audio data is then converted into text data using a speech recognition service such as the Google Cloud Speech-to-Text API. The text data is given a timestamp and is analyzed and formatted within the server. The formatted subtitle data is then sent back to the device.

[1056] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, and the analyzed emotional data is also displayed. This allows the user to understand not only the audio information but also the emotional state of the person.

[1057] For example, if a user plays a video titled "Introduction to Python," the device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio data and generates the text "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file and displayed as follows:

[1058] srt

[1059] 1

[1060] 00:00:01,000 --> 00:00:04,000

[1061] Hello, this is an introductory text for Python.

[1062] At the same time, the emotion engine analyzes the user's facial expressions and voice tone to generate emotion data such as "excitement" or "concentration." The device receives this subtitle data and emotion data and displays it in real time on the video playback screen. For example, it can be displayed as follows:

[1063] srt

[1064] 1

[1065] 00:00:01,000 --> 00:00:04,000

[1066] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[1067] An example of a prompt is, "Please explain the detailed process flow of a system that converts video audio into text in real time and displays it as subtitles, while also analyzing user emotions."

[1068] This invention allows users to enjoy subtitles in real time, deepening their understanding of video content, especially for the hearing impaired and in environments where multilingual support is required. Another major feature is that it can enrich the viewing experience by analyzing and displaying emotional information.

[1069] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1070] Step 1:

[1071] User plays a video

[1072] A user uses a video playback app to select a video and begin playing it.

[1073] Input: Select a video file and trigger playback.

[1074] Output: Video playback start signal.

[1075] Step 2:

[1076] Capture audio data

[1077] The device plays the audio track of the selected video using a MediaPlayer object, while simultaneously capturing the audio data in real time using an AudioRecord object.

[1078] Input: The audio track of the video file.

[1079] Output: Real-time captured audio data.

[1080] Specific operation: The captured audio data is temporarily stored in a buffer and divided into 1-second chunks.

[1081] Step 3:

[1082] Capturing Emotional Data

[1083] The device uses the user's camera footage and microphone input to analyze the user's facial expressions and voice tone using an emotion engine.

[1084] Input: User's camera video and audio input.

[1085] Output: Real-time analyzed emotion data.

[1086] Specific operation: By recognizing facial expressions from camera footage and analyzing the tone of voice from microphone input, an emotional state (such as "excitement" or "concentration") is generated.

[1087] Step 4:

[1088] Sending voice data to the server

[1089] The device asynchronously sends captured audio data to the server every second.

[1090] Input: Segmented audio data.

[1091] Output: Audio data sent to server completed.

[1092] Specific operation: The divided audio data is sent to the server using a protocol such as an HTTP request.

[1093] Step 5:

[1094] Saving audio data

[1095] The server saves the received audio data as a temporary file.

[1096] Input: Transmitted audio data.

[1097] Output: Audio data saved to a temporary file.

[1098] Specific operation: The audio data is saved as a file in WAV format or similar, and a timestamp is added to the file name.

[1099] Step 6:

[1100] Transcription of audio data

[1101] The server sends the stored audio data to the Google Cloud Speech-to-Text API and converts it into text data.

[1102] Input: Saved audio data file.

[1103] Output: Text data (with timestamp).

[1104] Specific operation: Calls the speech recognition service and converts the speech data into text format.

[1105] Step 7:

[1106] Subtitle data formatting

[1107] The server analyzes the converted text data and formats it into time-stamped subtitle data (such as .srt format).

[1108] Input: Text data generated by speech recognition.

[1109] Output: Formatted subtitle data.

[1110] Specific operation: Add appropriate line breaks and timestamps to the text data and format it into subtitle format.

[1111] Step 8:

[1112] Subtitle data transmission to device

[1113] The server sends the formatted subtitle data to the terminal.

[1114] Input: Formatted subtitle data.

[1115] Output: Subtitle data has been sent to the device.

[1116] Specific operation: Subtitle data is sent to the device using a protocol such as an HTTP request.

[1117] Step 9:

[1118] Displaying subtitle data and emotion data

[1119] The device analyzes the subtitle data received and the generated emotional data, and overlays it on the video playback screen in real time.

[1120] Input: Received subtitle data, parsed emotion data.

[1121] Output: Subtitles and emotions on the video playback screen.

[1122] Specific operation: Subtitle data and emotion data are synchronized based on timestamps and displayed in the appropriate position.

[1123] (Application example 2)

[1124] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1125] Conventional video playback systems have difficulty displaying subtitles in real time while a user is watching a video. Furthermore, they lack emotion recognition capabilities and are unable to display subtitles that take the user's emotional state into account, limiting the viewing experience. Real-time subtitle generation and emotional information display are particularly necessary for environments requiring support for the hearing impaired and multilingual support.

[1126] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for performing speech recognition and generating corresponding text data, means for assigning a timestamp to the generated text data and converting it into a subtitle format, and means for displaying emotion data generated in real time together with subtitles based on the timestamp. This makes it possible to generate subtitles in real time and display subtitles according to the user's emotional state.

[1127] A "terminal" is a device operated by a user that plays videos, extracts audio data, and recognizes emotions.

[1128] "Video data" refers to visual information contained in video content.

[1129] "Audio data" refers to the data of the audio track included in the video.

[1130] "Extraction" is the operation of extracting specific information (in this case, audio data) from a video.

[1131] "Transmit" means transferring data from one device to another device or server.

[1132] A "server" is a computer system that performs processes such as speech recognition and text generation.

[1133] "Speech recognition" is a technology that analyzes voice data and generates corresponding text data.

[1134] "Text data" is character information generated by voice recognition.

[1135] "Subtitle data" is text data to which a timestamp is added, and is displayed superimposed on video data.

[1136] A "timestamp" is time information that indicates a specific moment.

[1137] A "subtitle format" is a particular format in which text is displayed in video data.

[1138] "Facial expressions" are movements that show emotions and reactions on the user's face.

[1139] "Identification" is the act of analyzing video data and facial expression data to extract specific features and information.

[1140] An "emotional state" is information that describes the emotions a user is experiencing at a particular moment.

[1141] "Analysis" refers to examining data in detail to clarify its structure and meaning.

[1142] "Real-time" means that processing occurs immediately with little or no delay.

[1143] "Updating" means replacing data with new information.

[1144] This invention is a system consisting of a terminal and a server that transcribes the audio of videos watched by users in real time and displays them as subtitles, while also recognizing and displaying the user's emotional state.

[1145] System Configuration

[1146] Terminal

[1147] The device is a device that plays videos and extracts audio data. The scenario begins when a user selects a video using a video playback app on the device and starts playback. The device then captures audio data and performs emotion recognition. To do this, the device uses a MediaPlayer object to play the video's audio track and an AudioRecord object to capture audio data in real time. The device then uses a camera to capture the user's facial expressions and uses facial recognition technology to analyze their emotional state.

[1148] server

[1149] The server saves the received audio data in a temporary file and calls a speech recognition service to generate text data. The server then passes this audio data to a speech recognition service such as the Google Cloud Speech-to-Text API to generate corresponding text data. The server then adds a timestamp to the generated text data and converts it into subtitle format. The server then sends this text data to the device.

[1150] Hardware and Software

[1151] The device hardware includes a camera, microphone, and display. The software includes application software that runs on Android or iOS. On the server side, cloud services (such as Google Cloud or AWS) are used for voice recognition and sentiment analysis.

[1152] Data processing and calculation

[1153] The device captures audio data in real time. The audio data is divided into small chunks and sent sequentially to the server. The server converts the received audio data into text data, adds a timestamp, and formats it into subtitles. The device receives this subtitle data and displays it as an overlay on the video playback screen. At the same time, the device uses camera footage to analyze the user's facial expressions and update emotional data in real time.

[1154] Specific examples

[1155] For example, consider the case where a user is watching an educational video and the instructor says, "I will explain the basic syntax of Python." The device captures the audio and sends it to the server. The server converts the audio into text, generating the text "I will explain the basic syntax of Python." A timestamp is added to the text, and subtitle data is generated as follows:

[1156] 1

[1157] 00:00:01,000 --> 00:00:04,000

[1158] Learn about the basic syntax of Python

[1159] At the same time, the device captures the user's facial expressions with a camera and generates emotional data such as "excitement" or "concentration." This emotional data is also displayed with a timestamp, allowing the user to view subtitles containing emotional information in real time.

[1160] Prompt Sentence Examples

[1161] A real-time video subtitle generation and emotion recognition system. This application converts the audio of the video the user is watching into text, and simultaneously analyzes the user's emotions and adds them to the subtitles. Please write the following Python pseudocode:

[1162] Playing videos using OpenCV

[1163] Generate subtitles using the Google Cloud Speech-to-Text API

[1164] Recognizes and displays emotions using facial recognition and voice tone analysis

[1165] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1166] Step 1:

[1167] The user launches a video playback application on their device, selects a video to watch, and starts playing it, which starts playing the video using a MediaPlayer object.

[1168] Input: Video file

[1169] Output: Playing video

[1170] Step 2:

[1171] The device captures the audio track of the video in real time by using an AudioRecord object to obtain audio data in real time.

[1172] Input: Video currently playing

[1173] Output: Real-time captured audio data

[1174] Step 3:

[1175] The device divides the captured audio data into chunks of a fixed size and sends them to the server sequentially. The transmission is performed asynchronously, for example, sending chunk data every second.

[1176] Input: Captured audio data

[1177] Output: Audio data chunks sent to the server

[1178] Step 4:

[1179] The server saves the received audio data chunks as temporary files and calls a speech recognition service (such as the Google Cloud Speech-to-Text API) to convert the audio data into text.

[1180] Input: Audio data chunks

[1181] Output: Text data

[1182] Step 5:

[1183] The server adds a timestamp to the generated text data and converts it into a subtitle format, for example, generating SRT format subtitle data.

[1184] Input: Text data

[1185] Output: Subtitle data with timestamps

[1186] Step 6:

[1187] The server transmits the converted subtitle data to the terminal.

[1188] Input: Timestamped subtitle data

[1189] Output: Subtitle data sent to the device

[1190] Step 7:

[1191] The device overlays the received subtitle data on the video playback screen, and the subtitles are displayed at the appropriate time based on the timestamp.

[1192] Input: Timestamped subtitle data

[1193] Output: Overlaid subtitles

[1194] Step 8:

[1195] The device uses a camera to capture the user's facial expressions, analyzes the facial expression data, and performs real-time emotion recognition using an emotion recognition engine.

[1196] Input: Camera image

[1197] Output: Real-time analyzed emotion data

[1198] Step 9:

[1199] In order for the device to display the analyzed emotion data, the subtitle data and emotion data are integrated and displayed on the video playback screen.

[1200] Input: Analyzed emotion data, subtitle data

[1201] Output: Subtitles with emotional information

[1202] Step 10:

[1203] While watching a video, users can check the subtitles and emotional information generated in real time, providing a better viewing experience.

[1204] Input: subtitles with emotional information

[1205] Output: High-quality viewing experience

[1206] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1207] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1208] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1209] [Fourth embodiment]

[1210] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1211] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1212] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1213] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1214] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1216] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1217] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1218] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1219] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1220] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1221] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1222] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1223] The present invention uses a system that combines a terminal and a server to transcribe the audio in a video in real time and display it as subtitles. The specific operation of this system is described below.

[1224] System Overview

[1225] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[1226] Program structure and operation

[1227] User operations

[1228] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[1229] Terminal side processing

[1230] The device extracts the video's audio track in real time as soon as the video starts playing, using a MediaPlayer object to play the audio and an AudioRecord object to capture the audio data.

[1231] The extracted audio data is divided into chunks of a predetermined size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[1232] Server-side processing

[1233] The server saves the received audio data in a temporary file and sends it to a speech recognition service such as the Google Cloud Speech-to-Text API. The text data generated by the speech recognition is returned to the server with a timestamp.

[1234] The server parses the returned text data and formats it into a subtitle format, usually .srt or WebVTT, and sends the formatted subtitle data to the device.

[1235] Subtitle display on the device

[1236] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[1237] Specific examples

[1238] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[1239] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[1240] 1

[1241] 00:00:01,000 --> 00:00:04,000

[1242] Hello, this is an introductory text for Python.

[1243] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[1244] Advantages of the invention

[1245] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[1246] The processing flow will be explained below.

[1247] Step 1:

[1248] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[1249] Step 2:

[1250] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[1251] Step 3:

[1252] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[1253] Step 4:

[1254] The server saves the received voice data as a temporary file, which is then ready to be sent to the voice recognition server.

[1255] Step 5:

[1256] The server calls a speech recognition service, such as the Google Cloud Speech-to-Text API, and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[1257] Step 6:

[1258] The server analyzes the received text data and timestamp information and formats it into .srt or WebVTT format. Specifically, it defines the text and its start and end times.

[1259] Step 7:

[1260] The server sends the formatted subtitle data to the terminal as an HTTP response.

[1261] Step 8:

[1262] The device analyzes the subtitle data received and processes it to display it in a TextView object. The subtitle display is implemented using Handlers and Runnables so that it is updated in real time based on timestamps.

[1263] Step 9:

[1264] When a user watches a video, the device overlays subtitles onto the video in real time, allowing the user to understand the video content not only audibly but also visually.

[1265] Example 1

[1266] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1267] Video content is consumed daily, but people with hearing impairments and those in noisy environments often find it difficult to understand the audio. Furthermore, when watching videos in foreign languages, there is a demand for a function that displays audio as text data. However, existing systems lack real-time capabilities and are limited in their use due to limitations on the devices and services they can be used with, resulting in a lack of flexibility. Furthermore, delays in the display of subtitles can degrade the user experience.

[1268] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1269] In this invention, the server includes means for performing speech recognition processing and generating corresponding character data, means for formatting the generated character data into a subtitle format and attaching a timestamp, and means for transmitting the formatted subtitle data to the terminal, thereby enabling accurate subtitle display in real time and improving the user experience.

[1270] A "terminal" is a device that plays video, extracts audio, and sends it to a server.

[1271] "Audio signal" refers to data captured in real time from audio data within a video.

[1272] "Real time" means that processing, from audio extraction to subtitle display, is carried out almost simultaneously.

[1273] The "predetermined size" is the unit by which the audio signal is divided into a specified interval (for example, every second).

[1274] "Asynchronous" means that audio data is transmitted in real time, but each transmission is carried out independently.

[1275] The "server" is a device that receives an audio signal, performs voice recognition processing, generates subtitle data, and transmits it to a terminal.

[1276] "Speech recognition processing" is a technology that converts voice signals into text data.

[1277] "Character data" is text-format data generated by voice recognition.

[1278] A "subtitle format" is a specific format (e.g., .srt or WebVTT) used to display audio in a video as text.

[1279] A "timestamp" is information that is added to character data and indicates the exact timing of display.

[1280] "Overlay display" is a method of displaying content over the video playback screen.

[1281] The present invention uses a system that combines a terminal and a server to transcribe audio from a video in real time and display the transcribed audio as subtitles. An embodiment of this system will be described in detail below.

[1282] System Overview

[1283] This system transcribes the audio in a video and displays it as subtitles when the user plays the video on a device such as a smartphone, allowing people with hearing impairments and users in environments where it is difficult to hear the audio to understand the audio content of the video.

[1284] User operations

[1285] When a user selects a video using a video playback app on their device, such as YouTube, and starts playing it, the system works in the background to extract audio and generate subtitles.

[1286] Terminal side processing

[1287] The device extracts the audio signal of the video in real time as soon as the video starts playing. To do this, it uses a MediaPlayer object to play the audio and an AudioRecord object to capture the audio signal. The captured audio signal is divided into chunks of a specified size and sent to the server asynchronously. Data is sent to the server every second.

[1288] Server-side processing

[1289] The server saves the received audio signal to a temporary file and sends the data to a speech recognition service (e.g., Google Cloud Speech-to-Text API). The text data generated by speech recognition is returned to the server with a timestamp. The server parses the returned text data and formats it into a subtitle format (e.g., .srt, WebVTT). The formatted subtitle data is then sent to the device.

[1290] Subtitle display on the device

[1291] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, allowing the user to view the subtitles in real time.

[1292] Specific examples

[1293] For example, consider the case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio signal and generates text data such as "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file as follows:

[1294] 1

[1295] 00:00:01,000 --> 00:00:04,000

[1296] Hello, this is an introductory text for Python.

[1297] The device receives this subtitle data and displays it on the video playback screen, allowing users to view the subtitles in real time.

[1298] Prompt Sentence Examples

[1299] Prompt for explaining a program that uses generative AI models to generate real-time subtitles:

[1300] "Please tell me the process flow of a program that captures the audio of a video being played on the YouTube app and generates subtitles in real time using the Google Cloud Speech-to-Text API."

[1301] Advantages of the invention

[1302] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, the system configuration is simple, and by utilizing existing cloud services, it achieves both ease of implementation and high accuracy.

[1303] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1304] Step 1:

[1305] The user plays a video on their device. The input is the user selecting the video they want to play, and the output is the video starting to play on the device. Specifically, the user selects a video using the YouTube app or similar and presses the play button.

[1306] Step 2:

[1307] When the device starts playing the video, it also starts playing the audio signal using a MediaPlayer object. The input is the video file, and the output is the audio signal played in real time. Specifically, the MediaPlayer processes the audio signal and plays the audio.

[1308] Step 3:

[1309] The device uses an AudioRecord object to capture audio signals in real time. The input is the audio signal played by the MediaPlayer, and the output is the captured audio signal. Specifically, AudioRecord captures the audio signal and stores it in memory.

[1310] Step 4:

[1311] The device divides the captured audio signal into chunks of a specified size and sends them asynchronously to the server. The input is the captured audio signal, and the output is the audio signal data divided into chunks of a specified size (e.g., every second). Specifically, the audio signal is divided into chunks every second and sent to the server over the network.

[1312] Step 5:

[1313] The server saves the received audio signal in a temporary file. The input is the audio signal data sent from the terminal, and the output is the audio data saved in the temporary file. Specifically, the server saves the received audio signal as temp.wav.

[1314] Step 6:

[1315] The server sends the audio signal stored in a temporary file to the speech recognition service. The input is the temporary file of the stored audio signal, and the output is a request to the speech recognition service. Specifically, the server sends the audio file to the Google Cloud Speech-to-Text API and requests text data.

[1316] Step 7:

[1317] The speech recognition service analyzes the audio signal and generates corresponding text data. The input is the transmitted audio file, and the output is the generated text data. Specifically, the speech recognition service analyzes the audio signal and returns text data such as "Hello, this is an introductory text for Python."

[1318] Step 8:

[1319] The server adds a timestamp to the generated text data and formats it into subtitle format. The input is the text data returned from the speech recognition service, and the output is subtitle format data with a timestamp. Specifically, the text data is formatted into the following .srt format:

[1320] 1

[1321] 00:00:01,000 --> 00:00:04,000

[1322] Hello, this is an introductory text for Python.

[1323] Step 9:

[1324] The server sends the formatted subtitle data to the terminal. The input is time-stamped subtitle data, and the output is an API response to the terminal. Specifically, the server sends the formatted subtitle data to the terminal as an HTTP response.

[1325] Step 10:

[1326] The device analyzes the subtitle data received from the server and displays it as an overlay on the video playback screen. The input is the subtitle data sent from the server, and the output is real-time subtitles displayed on the video screen. Specifically, the device parses the received .srt file and displays the subtitles based on the timestamp. The user can view the subtitles in real time.

[1327] (Application example 1)

[1328] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1329] Current video streaming services do not adequately provide the functionality to display subtitles of audio content in real time while watching. This makes it difficult for the hearing impaired and users in multilingual environments to understand video content. In addition, the functionality available varies depending on the device, making it difficult to provide a consistent viewing experience for all users.

[1330] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1331] In this invention, the server includes means for performing speech recognition using received voice data and generating corresponding text data, means for assigning a timestamp to the generated text data, and means for converting the time-stamped text data into a subtitle format, thereby enabling subtitles to be generated and displayed in real time on a variety of playback devices.

[1332] A "terminal" is a device that extracts audio data, communicates with a server to receive text data, and displays the text data in a format that corresponds to the video data.

[1333] "Video data" refers to data of video content that a user views.

[1334] "Audio data" refers to data that indicates the audio portion included in the video data.

[1335] A "server" is a computer system that receives voice data sent from a terminal and performs voice recognition and generates text data.

[1336] "Speech recognition" is a process of converting received voice data into text data.

[1337] "Text data" is character string information generated by speech recognition.

[1338] A "timestamp" is time information added to text data, and indicates the timing of displaying subtitles.

[1339] The "subtitle format" is a data format for appropriately formatting the generated text data and displaying it in correspondence with the video data.

[1340] A "video distribution service" is an online service that distributes video content over the Internet.

[1341] "Real-time subtitle generation" is a function that instantly converts audio data into text data and displays it while a video is playing.

[1342] "A wide variety of playback devices" refers to various types of devices, such as smartphones, smart glasses, and head-mounted displays.

[1343] The present invention provides a system for generating and displaying subtitles in real time for video distribution services. This system is configured by combining a terminal, a server, and a voice recognition service.

[1344] System Configuration

[1345] 1. Device:

[1346] The hardware used includes smartphones, smart glasses, and head-mounted displays.

[1347] Audio data is extracted from the video data simultaneously with the video playback.

[1348] To capture audio data, on a smartphone, MediaPlayer (Android) or AVPlayer (iOS) is used, and AudioRecord (Android) or AVAudioRecorder (iOS) is used to record the audio data.

[1349] 2. Server:

[1350] Receives voice data sent from the terminal and performs voice recognition.

[1351] The Google Cloud Speech-to-Text API is used as the voice recognition service.

[1352] The text data generated by speech recognition is time-stamped and formatted into .srt or WebVTT format.

[1353] 3. Data flow:

[1354] When the device starts playing the video, it extracts the audio data and divides it into chunks of a specified size (e.g., 1 second).

[1355] The device asynchronously sends this to the server, which temporarily stores the received audio data and sends it to the Google Cloud Speech-to-Text API.

[1356] It receives the text data returned as a result of speech recognition, adds a timestamp to it, and formats it into subtitle format.

[1357] The formatted subtitle data is sent to the terminal, which receives it and displays it as subtitles.

[1358] Specific examples

[1359] Consider a specific example where a user is watching a movie on a smartphone using a video streaming service. When the user starts playing the video, the device captures the audio data and sends it to a server. The server then sends the received audio data to a speech recognition service, which generates text data such as "Once upon a time...". The generated text data is then given a timestamp and formatted in .srt format. This subtitle data is then sent to the device and displayed on the video screen in real time.

[1360] Prompt Sentence Examples

[1361] Develop an application that captures the audio track while a user is watching a movie on a video streaming service on their smartphone, converts it into text using the Google Cloud Speech-to-Text API, and displays it as real-time subtitles.

[1362] This system makes it possible to generate and display subtitles in real time on a variety of devices for video streaming services, thereby providing a consistent, high-quality viewing experience for users with hearing impairments and those in multilingual environments.

[1363] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1364] Step 1:

[1365] A user launches a video streaming service application on a device (smartphone, smart glasses, head-mounted display) and starts playing a video. At this point, the audio data extraction process begins along with the playback of the video data. Specifically, the device uses MediaPlayer (Android) or AVPlayer (iOS) to play the audio track of the video data in real time, and simultaneously captures the audio data using AudioRecord (Android) or AVAudioRecorder (iOS).

[1366] Input: Video playback start command, video data

[1367] Output: Extracted audio data

[1368] Step 2:

[1369] The extracted audio data is divided into chunks of a specified size (e.g., 1 second). The device sends this audio data to the server asynchronously. The division and transmission of the audio data is performed continuously, and continues without interruption while the user is watching the video. This maintains real-time performance.

[1370] Input: Extracted audio data

[1371] Output: Audio data chunks divided into predetermined sizes

[1372] Step 3:

[1373] The server temporarily stores the received audio data chunks and sends them to a speech recognition service such as the Google Cloud Speech-to-Text API, which converts the audio data into text data.

[1374] Input: Audio data chunks divided into predetermined sizes

[1375] Output: Generated text data

[1376] Step 4:

[1377] The server appends a timestamp to the text data returned by the speech recognition service and formats it into a subtitle format such as .srt or WebVTT. During this formatting process, the text data and the timestamp are formatted accordingly.

[1378] Input: Generated text data

[1379] Output: Subtitle data with timestamps

[1380] Step 5:

[1381] The server sends the formatted subtitle data to the device. The device receives this data and displays it as an overlay on the video playback screen. The display timing is based on the timestamp in real time, allowing the user to visually understand the audio content.

[1382] Input: Formatted subtitle data

[1383] Output: Display subtitles on the video playback screen

[1384] Through these processing steps, the video streaming service application can transcribe the audio of the video being watched by the user in real time and display it as subtitles, making it easier for users with hearing impairments and those in multilingual environments to understand the video content.

[1385] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1386] This invention combines a system that transcribes the audio of video content in real time and displays it as subtitles with an emotion engine, recognizing the user's emotional state and displaying it together with the subtitle data, thereby providing video viewers with richer information.

[1387] System Overview

[1388] This system consists of a terminal and a server. The terminal is operated by the user and is responsible for playing videos, extracting audio data, and recognizing emotions. The server, on the other hand, is responsible for voice recognition and generating subtitles. Furthermore, an emotion engine recognizes the user's emotions and displays the results along with the subtitle data.

[1389] Program structure and operation

[1390] User operations

[1391] The user selects a video using a video playback app on their device and starts playing it, while the system works in the background, extracting audio and recognizing emotions.

[1392] Terminal side processing

[1393] When the device starts playing the video, it uses the MediaPlayer object to play the video's audio track, and simultaneously captures the audio data in real time using the AudioRecord object. The captured audio data is divided into chunks of a specified size and sent to the server. The transmission is performed asynchronously, with data being sent to the server every second.

[1394] The emotion engine also analyzes the user's facial expressions and tone of voice using the user's camera footage and audio signals. The analyzed emotion data is updated in real time.

[1395] Server-side processing

[1396] The server saves the received audio data to a temporary file, then calls a speech recognition service such as the Google Cloud Speech-to-Text API and sends the saved audio file. The speech recognition service converts the audio file into text data and returns a response with a timestamp.

[1397] The server parses the returned text data and formats it into subtitles, which are then sent to the device.

[1398] Subtitle display on the device

[1399] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp. The analyzed emotional data is also displayed at the same time. This allows the user to understand not only the audio information but also the emotional state.

[1400] Specific examples

[1401] For example, consider a case where a user plays a video titled "Introduction to Python." The device captures the audio as soon as the video starts playing and sends it to the server in real time.

[1402] The server recognizes the received voice data and generates the text "Hello, this is an introductory text for Python." The generated text is formatted into a .srt file as follows:

[1403] 1

[1404] 00:00:01,000 --> 00:00:04,000

[1405] Hello, this is an introductory text for Python.

[1406] At the same time, the emotion engine analyzes the user's facial expressions and tone of voice to generate emotion data such as "excitement" or "concentration." For example, the following display is generated:

[1407] 1

[1408] 00:00:01,000 --> 00:00:04,000

[1409] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[1410] The device receives this subtitle data and emotion data and displays them on the video playback screen, allowing the user to view the emotion information along with the subtitles in real time.

[1411] Advantages of the invention

[1412] This invention allows users to enjoy subtitles in real time, enabling accurate understanding of video content, even for the hearing impaired and in environments requiring multilingual support. Furthermore, analyzing and displaying the user's emotional information can further enrich the viewing experience. Another major advantage is the simple system configuration, which combines ease of implementation with high accuracy by utilizing existing cloud services.

[1413] The processing flow will be explained below.

[1414] Step 1:

[1415] The user selects a video in the video playback app and presses the play button. The device obtains the video playback URL and starts playing the video using MediaPlayer.

[1416] Step 2:

[1417] The device uses AudioRecord to capture audio data in real time while the video is playing, and stores the captured audio data in a buffer.

[1418] Step 3:

[1419] To perform real-time emotion recognition, the device captures the user's face with a camera and the user's voice tone with a microphone, and these data are analyzed by the emotion engine.

[1420] Step 4:

[1421] The device splits the audio data in the buffer into chunks every second and sends them to the server using an HTTP POST request.

[1422] Step 5:

[1423] The server saves the received voice data as a temporary file and sends it to a speech recognition service such as Google Cloud Speech-to-Text API, which converts the voice data into text data and returns a response with a timestamp.

[1424] Step 6:

[1425] The server analyzes the received text data and its timestamp information and formats it into subtitle format (for example, .srt format).

[1426] Step 7:

[1427] The server sends the formatted subtitle data to the terminal as an HTTP response.

[1428] Step 8:

[1429] The device analyzes the received text data and processes it to overlay it on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp.

[1430] Step 9:

[1431] The device displays the analyzed emotion data along with the text data, which includes the user's emotional state (e.g., sad, happy, etc.) analyzed by the emotion engine.

[1432] Step 10:

[1433] Users can view their own emotional state along with subtitles while watching videos in real time, improving the viewing experience and providing emotional feedback.

[1434] Example 2

[1435] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1436] Conventional subtitling systems primarily convert audio data into text data for display, and are unable to reflect the user's emotional information. Furthermore, they lack the ingenuity to further enrich the user's viewing experience, limiting the information they can provide to viewers. As a result, there is a need for a method to further deepen the understanding of video content, especially for the hearing impaired and in environments requiring multilingual support.

[1437] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for extracting audio data from video data at the terminal; means for transmitting the extracted audio data to the server; means for performing speech recognition using the audio data received at the server and generating corresponding text data; means for transmitting the generated text data to the terminal; means for displaying the received text data at the terminal in a form corresponding to the video data; means for analyzing the user's facial expression and tone of voice at the terminal and generating emotional data of the user; and means for displaying the generated emotional data together with the text data in a form corresponding to the video data. This further enriches the user's viewing experience and enables a deeper understanding of video content, particularly for the hearing impaired or in an environment requiring multilingual support.

[1438] A "terminal" is a device that is operated by a user and has the functions of playing back video data, extracting audio data, and generating emotion data.

[1439] "Video data" refers to data that includes visual information of moving image content, and is played back in synchronization with audio data.

[1440] "Audio data" refers to the audio included in the video data, and is sound information that can be heard by the user.

[1441] A "server" is a computing device that receives data sent from a terminal and performs voice recognition and data conversion.

[1442] "Speech recognition" is a technology that analyzes voice data and converts it into corresponding text data.

[1443] "Text data" refers to data containing character information obtained by voice recognition, and is displayed in correspondence with video data.

[1444] "Emotion data" is data that indicates the emotional state of the user, obtained by analyzing the user's facial expressions and tone of voice.

[1445] The "subtitle format" is a format in which a timestamp is added to text data and displayed in synchronization with video data.

[1446] "Platform" means a system that includes the underlying software and hardware for performing video playback, audio capture, and sentiment analysis.

[1447] This system transcribes the audio of video content in real time, displays it as subtitles, and simultaneously recognizes the user's emotional state and displays it together with the subtitle data. This system consists of a terminal operated by the user and a server that processes the data.

[1448] A device is a device on which a user selects a video using a video playback app and starts playback. The device is equipped with a MediaPlayer object, which plays the video's audio track, and simultaneously captures audio data in real time using an AudioRecord object. The captured audio data is stored in a buffer and then split into 1-second chunks and sent asynchronously to the server.

[1449] The device also comes equipped with an emotion engine that uses camera footage and microphone input to analyze facial expressions and vocal tone, generating emotion data that is updated in real time.

[1450] The server receives the audio data sent from the device and saves it as a temporary file. This saved audio data is then converted into text data using a speech recognition service such as the Google Cloud Speech-to-Text API. The text data is given a timestamp and is analyzed and formatted within the server. The formatted subtitle data is then sent back to the device.

[1451] The device analyzes the received subtitle data and displays it as an overlay on the video playback screen. The subtitles are updated at the appropriate time based on the timestamp, and the analyzed emotional data is also displayed. This allows the user to understand not only the audio information but also the emotional state of the person.

[1452] For example, if a user plays a video titled "Introduction to Python," the device captures the audio as soon as the video starts playing and sends it to the server in real time. The server then performs speech recognition on the received audio data and generates the text "Hello, this is the text for Introduction to Python." The generated text is formatted into a .srt file and displayed as follows:

[1453] srt

[1454] 1

[1455] 00:00:01,000 --> 00:00:04,000

[1456] Hello, this is an introductory text for Python.

[1457] At the same time, the emotion engine analyzes the user's facial expressions and voice tone to generate emotion data such as "excitement" or "concentration." The device receives this subtitle data and emotion data and displays it in real time on the video playback screen. For example, it can be displayed as follows:

[1458] srt

[1459] 1

[1460] 00:00:01,000 --> 00:00:04,000

[1461] Hello, this is an introductory textbook on Python. [Emotion: Excitement]

[1462] An example of a prompt is, "Please explain the detailed process flow of a system that converts video audio into text in real time and displays it as subtitles, while also analyzing user emotions."

[1463] This invention allows users to enjoy subtitles in real time, deepening their understanding of video content, especially for the hearing impaired and in environments where multilingual support is required. Another major feature is that it can enrich the viewing experience by analyzing and displaying emotional information.

[1464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1465] Step 1:

[1466] User plays a video

[1467] A user uses a video playback app to select a video and begin playing it.

[1468] Input: Select a video file and trigger playback.

[1469] Output: Video playback start signal.

[1470] Step 2:

[1471] Capture audio data

[1472] The device plays the audio track of the selected video using a MediaPlayer object, while simultaneously capturing the audio data in real time using an AudioRecord object.

[1473] Input: The audio track of the video file.

[1474] Output: Real-time captured audio data.

[1475] Specific operation: The captured audio data is temporarily stored in a buffer and divided into 1-second chunks.

[1476] Step 3:

[1477] Capturing Emotional Data

[1478] The device uses the user's camera footage and microphone input to analyze the user's facial expressions and voice tone using an emotion engine.

[1479] Input: User's camera video and audio input.

[1480] Output: Real-time analyzed emotion data.

[1481] Specific operation: By recognizing facial expressions from camera footage and analyzing the tone of voice from microphone input, an emotional state (such as "excitement" or "concentration") is generated.

[1482] Step 4:

[1483] Sending voice data to the server

[1484] The device asynchronously sends captured audio data to the server every second.

[1485] Input: Segmented audio data.

[1486] Output: Audio data sent to server completed.

[1487] Specific operation: The divided audio data is sent to the server using a protocol such as an HTTP request.

[1488] Step 5:

[1489] Saving audio data

[1490] The server saves the received audio data as a temporary file.

[1491] Input: Transmitted audio data.

[1492] Output: Audio data saved to a temporary file.

[1493] Specific operation: The audio data is saved as a file in WAV format or similar, and a timestamp is added to the file name.

[1494] Step 6:

[1495] Transcription of audio data

[1496] The server sends the stored audio data to the Google Cloud Speech-to-Text API and converts it into text data.

[1497] Input: Saved audio data file.

[1498] Output: Text data (with timestamp).

[1499] Specific operation: Calls the speech recognition service and converts the speech data into text format.

[1500] Step 7:

[1501] Subtitle data formatting

[1502] The server analyzes the converted text data and formats it into time-stamped subtitle data (such as .srt format).

[1503] Input: Text data generated by speech recognition.

[1504] Output: Formatted subtitle data.

[1505] Specific operation: Add appropriate line breaks and timestamps to the text data and format it into subtitle format.

[1506] Step 8:

[1507] Subtitle data transmission to device

[1508] The server sends the formatted subtitle data to the terminal.

[1509] Input: Formatted subtitle data.

[1510] Output: Subtitle data has been sent to the device.

[1511] Specific operation: Subtitle data is sent to the device using a protocol such as an HTTP request.

[1512] Step 9:

[1513] Displaying subtitle data and emotion data

[1514] The device analyzes the subtitle data received and the generated emotional data, and overlays it on the video playback screen in real time.

[1515] Input: Received subtitle data, parsed emotion data.

[1516] Output: Subtitles and emotions on the video playback screen.

[1517] Specific operation: Subtitle data and emotion data are synchronized based on timestamps and displayed in the appropriate position.

[1518] (Application example 2)

[1519] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] Conventional video playback systems have difficulty displaying subtitles in real time while a user is watching a video. Furthermore, they lack emotion recognition capabilities and are unable to display subtitles that take the user's emotional state into account, limiting the viewing experience. Real-time subtitle generation and emotional information display are particularly necessary for environments requiring support for the hearing impaired and multilingual support.

[1521] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for performing speech recognition and generating corresponding text data, means for assigning a timestamp to the generated text data and converting it into a subtitle format, and means for displaying emotion data generated in real time together with subtitles based on the timestamp. This makes it possible to generate subtitles in real time and display subtitles according to the user's emotional state.

[1522] A "terminal" is a device operated by a user that plays videos, extracts audio data, and recognizes emotions.

[1523] "Video data" refers to visual information contained in video content.

[1524] "Audio data" refers to the data of the audio track included in the video.

[1525] "Extraction" is the operation of extracting specific information (in this case, audio data) from a video.

[1526] "Transmit" means transferring data from one device to another device or server.

[1527] A "server" is a computer system that performs processes such as speech recognition and text generation.

[1528] "Speech recognition" is a technology that analyzes voice data and generates corresponding text data.

[1529] "Text data" is character information generated by voice recognition.

[1530] "Subtitle data" is text data to which a timestamp is added, and is displayed superimposed on video data.

[1531] A "timestamp" is time information that indicates a specific moment.

[1532] A "subtitle format" is a particular format in which text is displayed in video data.

[1533] "Facial expressions" are movements that show emotions and reactions on the user's face.

[1534] "Identification" is the act of analyzing video data and facial expression data to extract specific features and information.

[1535] An "emotional state" is information that describes the emotions a user is experiencing at a particular moment.

[1536] "Analysis" refers to examining data in detail to clarify its structure and meaning.

[1537] "Real-time" means that processing occurs immediately with little or no delay.

[1538] "Updating" means replacing data with new information.

[1539] This invention is a system consisting of a terminal and a server that transcribes the audio of videos watched by users in real time and displays them as subtitles, while also recognizing and displaying the user's emotional state.

[1540] System Configuration

[1541] Terminal

[1542] The device is a device that plays videos and extracts audio data. The scenario begins when a user selects a video using a video playback app on the device and starts playback. The device then captures audio data and performs emotion recognition. To do this, the device uses a MediaPlayer object to play the video's audio track and an AudioRecord object to capture audio data in real time. The device then uses a camera to capture the user's facial expressions and uses facial recognition technology to analyze their emotional state.

[1543] server

[1544] The server saves the received audio data in a temporary file and calls a speech recognition service to generate text data. The server then passes this audio data to a speech recognition service such as the Google Cloud Speech-to-Text API to generate corresponding text data. The server then adds a timestamp to the generated text data and converts it into subtitle format. The server then sends this text data to the device.

[1545] Hardware and Software

[1546] The device hardware includes a camera, microphone, and display. The software includes application software that runs on Android or iOS. On the server side, cloud services (such as Google Cloud or AWS) are used for voice recognition and sentiment analysis.

[1547] Data processing and calculation

[1548] The device captures audio data in real time. The audio data is divided into small chunks and sent sequentially to the server. The server converts the received audio data into text data, adds a timestamp, and formats it into subtitles. The device receives this subtitle data and displays it as an overlay on the video playback screen. At the same time, the device uses camera footage to analyze the user's facial expressions and update emotional data in real time.

[1549] Specific examples

[1550] For example, consider the case where a user is watching an educational video and the instructor says, "I will explain the basic syntax of Python." The device captures the audio and sends it to the server. The server converts the audio into text, generating the text "I will explain the basic syntax of Python." A timestamp is added to the text, and subtitle data is generated as follows:

[1551] 1

[1552] 00:00:01,000 --> 00:00:04,000

[1553] Learn about the basic syntax of Python

[1554] At the same time, the device captures the user's facial expressions with a camera and generates emotional data such as "excitement" or "concentration." This emotional data is also displayed with a timestamp, allowing the user to view subtitles containing emotional information in real time.

[1555] Prompt Sentence Examples

[1556] A real-time video subtitle generation and emotion recognition system. This application converts the audio of the video the user is watching into text, and simultaneously analyzes the user's emotions and adds them to the subtitles. Please write the following Python pseudocode:

[1557] Playing videos using OpenCV

[1558] Generate subtitles using the Google Cloud Speech-to-Text API

[1559] Recognizes and displays emotions using facial recognition and voice tone analysis

[1560] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1561] Step 1:

[1562] The user launches a video playback application on their device, selects a video to watch, and starts playing it, which starts playing the video using a MediaPlayer object.

[1563] Input: Video file

[1564] Output: Playing video

[1565] Step 2:

[1566] The device captures the audio track of the video in real time by using an AudioRecord object to obtain audio data in real time.

[1567] Input: Video currently playing

[1568] Output: Real-time captured audio data

[1569] Step 3:

[1570] The device divides the captured audio data into chunks of a fixed size and sends them to the server sequentially. The transmission is performed asynchronously, for example, sending chunk data every second.

[1571] Input: Captured audio data

[1572] Output: Audio data chunks sent to the server

[1573] Step 4:

[1574] The server saves the received audio data chunks as temporary files and calls a speech recognition service (such as the Google Cloud Speech-to-Text API) to convert the audio data into text.

[1575] Input: Audio data chunks

[1576] Output: Text data

[1577] Step 5:

[1578] The server adds a timestamp to the generated text data and converts it into a subtitle format, for example, generating SRT format subtitle data.

[1579] Input: Text data

[1580] Output: Subtitle data with timestamps

[1581] Step 6:

[1582] The server transmits the converted subtitle data to the terminal.

[1583] Input: Timestamped subtitle data

[1584] Output: Subtitle data sent to the device

[1585] Step 7:

[1586] The device overlays the received subtitle data on the video playback screen, and the subtitles are displayed at the appropriate time based on the timestamp.

[1587] Input: Timestamped subtitle data

[1588] Output: Overlaid subtitles

[1589] Step 8:

[1590] The device uses a camera to capture the user's facial expressions, analyzes the facial expression data, and performs real-time emotion recognition using an emotion recognition engine.

[1591] Input: Camera image

[1592] Output: Real-time analyzed emotion data

[1593] Step 9:

[1594] In order for the device to display the analyzed emotion data, the subtitle data and emotion data are integrated and displayed on the video playback screen.

[1595] Input: Analyzed emotion data, subtitle data

[1596] Output: Subtitles with emotional information

[1597] Step 10:

[1598] While watching a video, users can check the subtitles and emotional information generated in real time, providing a better viewing experience.

[1599] Input: subtitles with emotional information

[1600] Output: High-quality viewing experience

[1601] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1602] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1603] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1604] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1605] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1606] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1607] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1608] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1609] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1610] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1611] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1612] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1613] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1614] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1615] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1616] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1617] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1618] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1619] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1620] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1621] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1622] The following is further disclosed regarding the above embodiment.

[1623] (Claim 1)

[1624] means for extracting audio data from video data in the terminal;

[1625] means for transmitting the extracted voice data to a server;

[1626] a means for performing speech recognition using the received speech data in the server and generating corresponding text data;

[1627] means for transmitting the generated text data to a terminal;

[1628] means for displaying the received text data in a form corresponding to the video data in the terminal;

[1629] A system including:

[1630] (Claim 2)

[1631] means for dividing the extracted audio data into a plurality of chunks at the terminal;

[1632] means for sequentially transmitting the audio data divided into a plurality of chunks to a server;

[1633] The system of claim 1 further comprising:

[1634] (Claim 3)

[1635] means for adding a timestamp to the generated text data in the server;

[1636] A means for converting the time-stamped text data into a subtitle format;

[1637] The system of claim 1 further comprising:

[1638] "Example 1"

[1639] (Claim 1)

[1640] means for extracting an audio signal in real time at the terminal;

[1641] means for dividing the extracted audio signal into pieces of a predetermined size and transmitting the pieces to a server asynchronously;

[1642] a means for performing speech recognition processing in the server and generating corresponding character data;

[1643] means for formatting the generated text data into a subtitle format and adding a timestamp;

[1644] means for transmitting formatted subtitle data to a terminal;

[1645] means for overlaying and displaying the received subtitle data on a video playback screen in the terminal;

[1646] A system including:

[1647] (Claim 2)

[1648] 2. The system according to claim 1, wherein the voice signals extracted at the terminal are sequentially transmitted to the server.

[1649] (Claim 3)

[1650] 2. The system according to claim 1, wherein the subtitle data is updated and displayed in real time based on a timestamp assigned by the server.

[1651] "Application Example 1"

[1652] (Claim 1)

[1653] means for extracting audio data from video data in the terminal;

[1654] means for transmitting the extracted voice data to a server;

[1655] a means for performing speech recognition using the received speech data in the server and generating corresponding text data;

[1656] means for transmitting the generated text data to a terminal;

[1657] means for displaying the received text data in a form corresponding to the video data in the terminal;

[1658] A method that works on video streaming services and generates and displays subtitles in real time to support a wide variety of playback devices.

[1659] A system including:

[1660] (Claim 2)

[1661] means for dividing the extracted audio data into a plurality of chunks at the terminal;

[1662] means for sequentially transmitting the audio data divided into a plurality of chunks to a server;

[1663] The system of claim 1 further comprising:

[1664] (Claim 3)

[1665] means for adding a timestamp to the generated text data in the server;

[1666] A means for converting the time-stamped text data into a subtitle format;

[1667] a means for appropriately displaying the received subtitle data in the terminal based on a timestamp;

[1668] The system of claim 1 further comprising:

[1669] Example 2: Combining Emotion Engines

[1670] (Claim 1)

[1671] means for extracting audio data from video data in the terminal;

[1672] means for transmitting the extracted voice data to a server;

[1673] a means for performing speech recognition using the received speech data in the server and generating corresponding text data;

[1674] means for transmitting the generated text data to a terminal;

[1675] means for displaying the received text data in a form corresponding to the video data in the terminal;

[1676] means for analyzing a user's facial expression and tone of voice in the terminal and generating emotion data of the user;

[1677] means for displaying the generated emotion data together with text data in a form corresponding to the video data;

[1678] A system including:

[1679] (Claim 2)

[1680] means for dividing the extracted audio data into a plurality of chunks at the terminal;

[1681] means for sequentially transmitting the audio data divided into a plurality of chunks to a server;

[1682] The system of claim 1 further comprising:

[1683] (Claim 3)

[1684] means for adding a timestamp to the generated text data in the server;

[1685] A means for converting the time-stamped text data into a subtitle format;

[1686] means for transmitting the converted subtitle data to a terminal;

[1687] The system of claim 1 further comprising:

[1688] Application example 2 when combining emotion engines

[1689] (Claim 1)

[1690] means for extracting audio data from video data in the terminal;

[1691] means for transmitting the extracted voice data to a server;

[1692] a means for performing speech recognition using the received speech data in the server and generating corresponding text data;

[1693] means for transmitting the generated text data to a terminal;

[1694] means for displaying the received text data in a form corresponding to the video data in the terminal;

[1695] means for identifying facial expressions in the terminal to analyze the emotional state of the user;

[1696] a means for displaying the analyzed emotional state together with the subtitle data;

[1697] A system including:

[1698] (Claim 2)

[1699] means for dividing the extracted audio data into a plurality of chunks at the terminal;

[1700] means for sequentially transmitting the audio data divided into a plurality of chunks to a server;

[1701] 10. The system of claim 1, further comprising means for updating the emotional state from the user's facial expressions in real time.

[1702] (Claim 3)

[1703] means for adding a timestamp to the generated text data in the server;

[1704] A means for converting the time-stamped text data into a subtitle format;

[1705] 10. The system of claim 1, further comprising means for displaying emotion data generated in real time together with subtitles based on time stamps. [Explanation of symbols]

[1706] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for extracting audio data from video data in the terminal; means for transmitting the extracted voice data to a server; a means for performing speech recognition using the received speech data in the server and generating corresponding text data; means for transmitting the generated text data to a terminal; means for displaying the received text data in a form corresponding to the video data in the terminal; A system including:

2. means for dividing the extracted audio data into a plurality of chunks at the terminal; means for sequentially transmitting the audio data divided into a plurality of chunks to a server; The system of claim 1 further comprising:

3. means for adding a timestamp to the generated text data in the server; A means for converting the time-stamped text data into a subtitle format; The system of claim 1 further comprising:

Images