system

Abstract

We provide the system. [Solution] A means of analyzing audio data in real time and converting it into text data, A means for translating the aforementioned text data into multiple languages, A method for analyzing video data and proposing subtitle display styles, A means for automatically generating subtitles using the translated texts of the aforementioned multiple languages, A means to allow the user to edit the display style of the aforementioned subtitles, A means of displaying the aforementioned subtitles in real time during a live broadcast, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The creation and management of multilingual subtitles in videos and live broadcasts have traditionally been done manually, requiring a great deal of time and labor. Also, since the adjustment of subtitle styles to enhance the visual appeal to viewers also needs to be done manually, high-quality content production has become an issue. In particular, in the case of live broadcasts, real-time subtitle generation is required, and delays in work have become a serious problem.

Means for Solving the Problems

[0005] This invention provides a function to analyze audio data in real time and convert it into text, and a function to automatically generate subtitles by translating that text into multiple languages. Furthermore, by analyzing video data, it proposes subtitle styles such as color and font size according to the scene, and makes them editable by the user, thereby meeting diverse needs. In addition, by having a function to display these subtitles in real time even during live streaming, it enables rapid and efficient subtitle management.

[0006] "Audio data" refers to information that has been saved in digital format and made into a processable state.

[0007] "Real-time" refers to the ability to process an event at the very moment it occurs.

[0008] "Text data" refers to a collection of information expressed as characters or sentences.

[0009] Translation is the act of converting text written in one language into another language.

[0010] Subtitles are text displayed as additional information to a video, and are usually a written transcription of the audio.

[0011] "Video data" refers to a collection of visual information expressed as images or videos.

[0012] "Style" refers to the visual characteristics of subtitles, such as color, font size, and position.

[0013] "Live streaming" refers to video streaming technology used to deliver content in real time.

[0014] A "user" refers to the entity that operates the system and utilizes its results. [Brief explanation of the drawing]

[0015] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a tagged processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a tagged storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, a tagged communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention relates to a system that analyzes audio data in real time, translates it, and generates multilingual subtitles. The system aims to automate the entire process from audio input to subtitle display, enabling users to efficiently provide high-quality content.

[0037] The server first acquires audio data from video and live streams, performs noise reduction, and then converts it into text data using a speech recognition model. This enables real-time conversion from audio information to text information.

[0038] The acquired text data is translated by the server into the language specified by the user using a multilingual translation model. The translated text is then incorporated into a subtitle data structure within the server and managed along with information such as timestamps.

[0039] The device analyzes video data and suggests the optimal subtitle display style for each scene. This includes suggestions regarding subtitle color, font size, and position, supporting a visually appealing display for the user.

[0040] Users can view these suggestions via their device and customize them as needed. For example, users can adjust the font size to make subtitles more prominent in bright scenes.

[0041] Ultimately, the server overlays the edited subtitles onto the video or live stream in real time. This allows viewers to instantly see subtitles in different languages, enabling broadcasting that caters to a multinational audience.

[0042] This system is particularly useful in specific applications such as international conferences and online events targeting multinational audiences, enabling the provision of accurate multilingual information while simplifying user operation.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server acquires audio data from video and live streams, and generates clear audio data by removing noise through filtering.

[0046] Step 2:

[0047] The server inputs the denoised audio data into a speech recognition model and converts the auditory information into text data in real time. This text data is then sent to the subsequent translation process.

[0048] Step 3:

[0049] The server processes the converted text data through a multilingual translation model, enabling accurate translations into multiple specified languages. Each translation result is then ready for subtitle generation.

[0050] Step 4:

[0051] The server constructs timestamped subtitle data based on the translated text. This data is then formatted for use as subtitles.

[0052] Step 5:

[0053] The device analyzes the video data and suggests the optimal subtitle display style (color, font size, position, etc.) based on the scene's color tone and elements.

[0054] Step 6:

[0055] Users can review suggested subtitle styles through their device and manually customize the subtitle design as needed. Edits are immediately reflected in the system.

[0056] Step 7:

[0057] The server overlays user-subtitled or edited subtitles onto the video or live stream in real time, providing viewers with multilingual subtitles.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] Currently, providing multilingual video content in real time presents challenges in achieving high-quality translations and appropriate subtitles, resulting in delays and inaccuracies in providing information in multiple languages. Furthermore, subtitle formats may not suit viewers' visual preferences, potentially detracting from the viewing experience. To address these issues, a system is needed that instantly translates audio information and suggests the most suitable subtitle format based on the visual information.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing visual information and proposing a display format for the translated text information. This makes it possible to generate multilingual encoded subtitles in real time and provide them in a display format that suits visual preferences.

[0063] "Audio information" refers to sound signals transmitted through sound waves, including acoustic data such as human voices.

[0064] "Real-time" refers to a state where data processing and communication occur instantly, resulting in virtually no delay and immediate processing results.

[0065] "Textual information" refers to data that represents information transmitted through sound or sight as written characters in a language.

[0066] "Multiple languages" refers to a set of languages ​​with different linguistic systems, including various languages ​​used internationally.

[0067] Translation is the act of converting information expressed in one language into another language, thereby maintaining meaning while expressing equivalent content in a different linguistic system.

[0068] "Visual information" refers to data obtained through vision, including information such as videos and images.

[0069] "Display format" refers to the layout and style used when data or information is presented visually, and includes characteristics such as color, size, and position.

[0070] "Real-time composite display" refers to the process of instantly combining multiple data points and presenting them visually without delay.

[0071] To implement this invention, the system is configured as follows.

[0072] The server first acquires audio information from acoustic data collection devices or digital streaming services. This audio information is preprocessed using a noise reduction algorithm and then converted into text information using a speech recognition model (e.g., a speech recognition API). This conversion allows for the rapid and accurate conversion of information obtained from audio signals into text format.

[0073] Next, the server utilizes a multilingual translation model (e.g., a translation service API) to translate text information into multiple languages. The translated text information is managed appropriately in synchronization with the visual information. This process eliminates language barriers, allowing viewers to understand the content in their native language.

[0074] The terminal receives visual information from the video output device, analyzes that information, and automatically suggests the optimal display format. This includes settings such as color, font size, and placement, and is adjusted to provide the best possible display for the viewing environment and individual scenes.

[0075] Users can review and customize these suggestions via their devices. Specifically, they can adjust the subtitle display style to their preferences and needs by manipulating the device's user interface. This allows viewers to enjoy a visual experience tailored to their individual visual preferences.

[0076] Ultimately, the server integrates all settings and composites them into the live stream or video content in real time. This allows viewers to instantly understand information in multiple languages, enabling the provision of appropriate information to audiences with diverse cultural backgrounds.

[0077] As a concrete example, this system can be used in international online events to allow participants speaking different languages ​​to follow the discussion simultaneously in their respective languages. An example of a prompt would be, "Please translate the audio of this meeting into English, Spanish, and French in real time and display the subtitles." In response to this prompt, the server retrieves the audio information, translates it, and provides the information to each terminal in the appropriate display format.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The server acquires audio information from video streams and audio acquisition devices. The input is either an audio signal or streaming data, which the server records as an audio file. A noise reduction algorithm is applied to ensure the audio is clear. As a result, the server outputs audio data with the noise removed.

[0081] Step 2:

[0082] The server passes the denoised audio data to a speech recognition model for conversion to text. The input for this step is denoised audio data, and the server generates text data in real time using a speech recognition API. The output is text data converted from speech.

[0083] Step 3:

[0084] The server inputs the generated text data into a multilingual translation model and translates it into multiple languages ​​specified by the user. The input for this step is the text data obtained from speech recognition, and the server outputs the translated text data in each language using a translation service API.

[0085] Step 4:

[0086] The terminal receives video data and analyzes the visual information to help optimize subtitle display. The input for this step is video data, and the terminal uses a scene analysis algorithm to suggest a subtitle display style appropriate for the scene. The output is the settings (color, size, position) of the suggested display style.

[0087] Step 5:

[0088] The user reviews and customizes the suggested subtitle display style via the device's interface. The input is the suggested display style from the device, and the customized style settings are output based on the user's actions. During this process, the user can adjust font size and color, and correct the display position.

[0089] Step 6:

[0090] The server applies a user-customized subtitle display style to the video, generating a composite video in real time. The input for this step is translated text data and the customized display style, which the server integrates to output the final subtitled video. As a result, viewers can view multilingual subtitles in real time.

[0091] (Application Example 1)

[0092] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] International content distribution services require the real-time provision of subtitles in multiple languages ​​and the creation of customizable visual display formats for each viewer. However, current methods have limitations in translation accuracy and display flexibility, which can degrade the quality of the viewing experience. A solution is needed to address these issues and provide an efficient and high-quality multilingual subtitle service.

[0094] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0095] In this invention, the server includes means for analyzing an audio signal in real time and converting it into document data, means for converting the document data into multiple natural languages, and means for analyzing a video signal and proposing a subtitle display format. This enables the real-time generation of subtitles in various languages ​​and allows users to visually customize the subtitle display.

[0096] An "audio signal" is an electrical representation of sound information that is analyzed in real time and converted into text.

[0097] "Document data" refers to text information converted from audio signals, which serves as the basis for translation into multiple languages.

[0098] "Natural language" refers to the words and languages ​​that people use on a daily basis, and is the subject of document data translation.

[0099] A "video signal" is an electrical representation of visual information, and it is analyzed when proposing a subtitle display format.

[0100] Subtitles are a visual representation of the audio content within a video, and they are generated in real time.

[0101] "Display format" refers to the elements that determine the appearance of subtitles, and includes visual characteristics such as color, font size, and placement.

[0102] A "user" is an individual or group that can use the system to perform visual customizations.

[0103] A "server" is a computer device that performs audio signal analysis, translation, and subtitle generation, and forms the core of the system.

[0104] The system for implementing this invention mainly consists of a server and terminals. The central server analyzes speech signals in real time using Google® Cloud Speech-to-Text and converts them into document data. Subsequently, this document data is translated into multiple natural languages ​​using the Google Cloud Translation API, and translated documents are obtained. The translated document data is used as subtitles, and data management is performed for visual customization.

[0105] The device provides an interface to the user through an application developed with React Native, suggesting subtitle display formats. Through this interface, the user can freely customize the subtitle's color, font size, and placement. Finally, the customized subtitles are visualized in real time during communication via the device.

[0106] A concrete example is that on international anime streaming platforms, this system allows viewers to easily access real-time subtitles in different languages. This enables multilingual subtitled streaming, allowing users from multiple countries to enjoy content simultaneously.

[0107] An example of a prompt using a generative AI model is, "Search for the anime title by voice, translate the audio from Japanese to English, and display the subtitles." This allows users to fully understand content even in languages ​​other than their native tongue.

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The server acquires video and audio data from the streaming source in real time. It recognizes this data as an audio signal and applies a noise reduction filter to generate clear audio data. This provides high-quality input data for analysis.

[0111] Step 2:

[0112] The server analyzes the denoised audio signal using Google Cloud Speech-to-Text and converts it into document data. This document data is the text data necessary to accurately represent the content of the audio. This text data serves as the input for the translation process.

[0113] Step 3:

[0114] The server uses the Google Cloud Translation API to translate the document data generated in the previous step into the specified natural languages. This translation process generates document data corresponding to each language, preparing it for the next subtitle generation step.

[0115] Step 4:

[0116] The server generates subtitle data based on the translated document data. This subtitle data includes timestamps and document information specific to each language, and it needs to be displayed in sync with the video.

[0117] Step 5:

[0118] The device analyzes this subtitle data and suggests a visually appealing display format to the user. Customization options are provided to adjust colors, font size, and layout. Based on this information, the user decides on the display format.

[0119] Step 6:

[0120] Users select and modify the subtitle display format using the interface on their device. The selected display format will be used in the next real-time display.

[0121] Step 7:

[0122] The device ultimately overlays the user-customized subtitle data onto the video, completing the visual output. This allows viewers to see subtitles in their chosen language in real time.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention provides a voice and video analysis system that combines an emotion engine to recognize the user's emotional state in real time and dynamically adjust the subtitle display style accordingly. This system has a multilingual subtitle generation function and provides visually and emotionally optimized content, taking into account the video content and the user's emotions.

[0125] The server processes video and live-stream audio data in real time and converts it into text data through a speech recognition model. This text information is translated within the server, generating subtitle data in multiple languages. Furthermore, the server uses an emotion engine to analyze the user's emotions from camera footage and audio acquired from the user's device. This analysis information influences the display format of the subtitles.

[0126] The device analyzes video data and suggests subtitle styles, such as color, font size, and position, based on the scene. Furthermore, the device dynamically adjusts the subtitle style in response to the user's emotions based on emotion information obtained from the emotion engine. For example, if the user is surprised, the font size of the subtitles can be increased to highlight them.

[0127] Users can review suggested subtitle styles through their device's interface and make manual adjustments. The emotion engine automatically provides optimal subtitles tailored to the user's emotions, without requiring any user changes to the default settings.

[0128] Furthermore, the server can generate visual images and illustrations based on emotion recognition and display them along with subtitles as needed, thereby further enhancing the impact of the video. This makes it possible to provide viewers with an even more engaging media experience.

[0129] This system enables the creation of highly responsive content that takes audience emotions into account, particularly in the entertainment and education sectors, significantly improving the user experience.

[0130] The following describes the processing flow.

[0131] Step 1:

[0132] The server acquires audio data from video and live streams, applies noise reduction filters, and prepares it for use in speech recognition models.

[0133] Step 2:

[0134] The server inputs the denoised audio into a speech recognition model and converts it into text data in real time. This data is used as the basis for subtitle generation.

[0135] Step 3:

[0136] The server passes the converted text data through a multilingual translation model and translates it into the specified multiple languages. This generates multilingual text that can be used as subtitles.

[0137] Step 4:

[0138] The server inputs camera footage and audio from the user's device into the emotion engine to analyze the user's emotional state. This information is used for dynamic adjustment of subtitle display.

[0139] Step 5:

[0140] Based on the analysis results from the emotion engine, the server generates instructions to optimize the display color, font size, and position of the subtitles.

[0141] Step 6:

[0142] The device analyzes video data and suggests subtitle display styles appropriate to the scene. It also adjusts the style based on the user's emotions, incorporating the results of the emotion engine.

[0143] Step 7:

[0144] Users can review and edit suggested subtitle styles on their device's interface. Even if the user does nothing, the system automatically adjusts the subtitle display to the optimal setting.

[0145] Step 8:

[0146] The server overlays user-reviewed or edited subtitles, along with optimization results from the emotion engine, onto the video or live stream in real time and provides them to viewers.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] In video and audio content, there is a need to dynamically adjust subtitle styles in real time according to emotions so that viewers can emotionally empathize with the content, transcending language and cultural differences. However, current systems struggle to efficiently and effectively achieve this requirement. In particular, integrating multilingual translation, emotion analysis, and visual adjustments is difficult, resulting in a less-than-optimal user experience.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing the user's emotional state and dynamically adjusting the subtitle display style. This enables optimal subtitle display that responds to the viewer's emotions, achieving deep emotional resonance in international media content and dramatically improving the user experience.

[0152] "Audio information" refers to information that represents audio waveforms as digital data and is in a format that can be analyzed by speech recognition processes.

[0153] "Textual information" refers to data in text format obtained through the analysis of audio information.

[0154] "Translating into a language" is the process of converting written information from one language into another different language to convey its meaning.

[0155] "Visual information" refers to information that includes visual images and video data, and is used for the analysis of visual content.

[0156] "Subtitle display style" refers to the style and format of visually displayed text information, including elements such as color, font size, and position.

[0157] "Analyzing a user's emotional state" is the process of determining a user's emotions from their facial expressions, tone of voice, etc., and expressing that state quantitatively or qualitatively.

[0158] "Real-time display" refers to the process of immediately visualizing data after it has been generated or received.

[0159] "Visual information and shapes" refer to illustrations and graphic elements added to video information, which play a role in assisting the understanding of the content.

[0160] This invention is a system that analyzes audio and video data in real time and provides subtitles tailored to the user's emotions. It primarily functions through the interaction of a server, terminals, and users.

[0161] The server is responsible for converting speech information into text information using a speech recognition model. Specifically, a speech recognition API can be applied as the software. The converted text information is then translated into multiple languages ​​using multilingual translation software. In this process, a translation API is used to achieve efficient and accurate multilingual translation.

[0162] Furthermore, the server analyzes camera video and audio data transmitted from the user device through an emotion analysis engine to infer the user's emotional state. By utilizing an emotion recognition API, it becomes possible to determine emotions from the user's facial expressions and voice tone. This analysis result is used to dynamically adjust the display style of subtitles.

[0163] The device proposes subtitle styles based on translated subtitle data and sentiment analysis results sent from the server. Specifically, it uses video editing software to automatically generate a visual style optimized for the scene content and the user's emotional state. For example, in emotional scenes, highlighting using warm colors is suggested.

[0164] Users can review and manually adjust suggested display styles through the device interface. By customizing the style themselves, rather than relying on default settings, users can achieve a more personalized content presentation. Utilizing a generative AI model, users can input prompts such as "If surprised, change subtitles to red," enabling specific and intuitive customization.

[0165] This system has the potential to improve the user experience across various content sectors, including entertainment and education. Viewers can gain a new viewing experience where they can intuitively understand and emotionally empathize with content from different cultures and languages ​​in their own language.

[0166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0167] Step 1:

[0168] The server receives audio data and converts it into text data using a speech recognition model. The input is audio data, and the output is the converted text data. By using a speech recognition API, the audio waveform is analyzed and text data is generated in real time.

[0169] Step 2:

[0170] The server inputs the character data obtained in step 1 into the translation module and performs multilingual translation. The input is character data, and the output is text data translated into multiple languages. A translation API is applied to perform fast and accurate translation.

[0171] Step 3:

[0172] The server uses an emotion analysis engine to evaluate the user's emotional state based on camera footage and audio sent from the user's device. The input is the user's video and audio data, and the output is the analysis result indicating their emotional state. It utilizes an emotion recognition API to analyze the user's facial expressions and voice and determine the corresponding emotions.

[0173] Step 4:

[0174] The terminal selects a subtitle style based on the translated text and sentiment analysis results received from the server. The input is the translated text and sentiment analysis results, and the output is the proposed subtitle style. Video editing software is then used to determine appropriate colors, font sizes, and other elements.

[0175] Step 5:

[0176] The user reviews the subtitle style displayed on the device and makes adjustments as needed. The input is the suggested style, and the output is the final adjusted style. The user can customize the subtitles, such as adjusting colors and changing fonts, using an intuitive interface.

[0177] Step 6:

[0178] The server receives the final subtitle style, integrates it into the video, and delivers it to viewers in real time. The input is the final style and video data, and the output is the video with subtitles. The goal is to enhance the viewer experience by providing them with emotionally optimized video content.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] Conventional subtitling systems simply convert audio to text and display subtitles, making it difficult to provide interactive content that takes into account the viewer's emotional state. Therefore, there is a need to provide a richer and more engaging viewing experience by analyzing the user's emotions in real time and automatically adjusting the subtitle display style accordingly.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes means for analyzing the user's emotional state obtained from audio and video information, means for dynamically adjusting the subtitle display style according to the user's emotional state, and means for creating relevant visual information using a generative AI model and suggesting its display along with subtitles as needed. This makes it possible to interactively adjust the content display style according to the viewer's emotions and provide a more engaging and personalized viewing experience.

[0184] "Acoustic information" refers to speech and other auditory data that can be converted into textual information through analysis.

[0185] "Textual information" refers to data in text format obtained by analyzing acoustic information, and is used for translation and subtitle generation.

[0186] "Visual information" refers to visual data, and analyzing this data is useful for determining the display format of subtitles and generating visual information.

[0187] "User emotional state" refers to data that indicates the user's emotional response, analyzed based on indicators obtained from video and audio information.

[0188] "Dynamically adjusting subtitle display style" refers to a function that changes the font, color, size, and placement of subtitles in real time according to the user's emotional state.

[0189] A "generative AI model" refers to artificial intelligence technology that generates new data or images based on input data, and is also used to create visual information.

[0190] "Visual information" refers to illustrations and images created using generative AI models, which are added to video content.

[0191] Subtitles are a way of displaying audio in video content as text, and they can also be used to support multiple languages.

[0192] The system realizing this invention connects the user's device (such as a smartphone or smart glasses) with a server and operates based on acoustic and video information. Acoustic information is collected through the device's microphone, and the server converts it into text information. Speech recognition technology is used for the conversion, and real-time processing is performed using Python and TENSORFLOW®.

[0193] The server further translates the acquired text information into multiple languages ​​and automatically generates multilingual subtitles. In addition, it analyzes the user's emotional state from video and audio information using an emotion engine. OpenCV is used for video processing here. A function is incorporated to adjust the subtitle display style (font size, color, position, etc.) in real time according to the user's emotional state.

[0194] On the user's device, the generated subtitles are displayed in sync with the video. Additionally, a generative AI model is used to create related visual information, which is then displayed in conjunction with the subtitles. Specifically, generative models such as Stable Diffusion are utilized to dynamically generate illustrations and images that correspond to the user's emotional state.

[0195] For example, if the system detects a smile while a user is watching a comedy film, it can brighten the subtitles and add illustrations. This system allows subtitles to change according to the viewer's emotions while watching movies or videos, providing a personalized viewing experience.

[0196] An example of a prompt for the generation AI model would be, "Generate more colorful and humorous illustrations when the user is laughing."

[0197] This allows viewers to receive an optimized media experience that responds to their emotions.

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The device collects acoustic information. This acoustic information is acquired by the device's microphone. When acoustic information is received, the device transmits it to the server in real time.

[0201] Step 2:

[0202] The server converts acoustic information into text information. The server takes acoustic information as input and outputs it as text information using speech recognition technology. Specifically, it uses TensorFlow to run a speech recognition model and outputs the resulting text data.

[0203] Step 3:

[0204] The server translates the text information. It takes the text information as input and uses a translation API to generate translated text for multiple languages. The output translated text is used for automatic subtitle generation.

[0205] Step 4:

[0206] The device analyzes the user's emotional state from video information. It analyzes the video information input to the device using OpenCV or similar tools, detecting the user's facial expressions and movements to determine their emotional state. The analysis results are sent to the server as emotional data.

[0207] Step 5:

[0208] The server adjusts the subtitle display style based on emotional state. It takes emotional data as input, dynamically sets the font size, color, and position of the subtitles based on this data, and outputs the adjusted subtitles.

[0209] Step 6:

[0210] The user's device generates visual information using a generative AI model. It takes emotional data and prompt text as input, and uses generative AI models such as Stable Diffusion to generate relevant visual information, which is then displayed alongside the video content. The output visual information serves as an additional element that complements the video experience.

[0211] Step 7:

[0212] Users can view subtitles that respond to their emotions and manually adjust the display style as needed. Users can use the settings screen as input to modify the display style. This allows for an even more personalized viewing experience.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] This invention relates to a system that analyzes audio data in real time, translates it, and generates multilingual subtitles. The system aims to automate the entire process from audio input to subtitle display, enabling users to efficiently provide high-quality content.

[0230] The server first acquires audio data from video and live streams, performs noise reduction, and then converts it into text data using a speech recognition model. This enables real-time conversion from audio information to text information.

[0231] The acquired text data is translated by the server into the language specified by the user using a multilingual translation model. The translated text is then incorporated into a subtitle data structure within the server and managed along with information such as timestamps.

[0232] The device analyzes video data and suggests the optimal subtitle display style for each scene. This includes suggestions regarding subtitle color, font size, and position, supporting a visually appealing display for the user.

[0233] Users can view these suggestions via their device and customize them as needed. For example, users can adjust the font size to make subtitles more prominent in bright scenes.

[0234] Ultimately, the server overlays the edited subtitles onto the video or live stream in real time. This allows viewers to instantly see subtitles in different languages, enabling broadcasting that caters to a multinational audience.

[0235] This system is particularly useful in specific applications such as international conferences and online events targeting multinational audiences, enabling the provision of accurate multilingual information while simplifying user operation.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The server acquires audio data from video and live streams, and generates clear audio data by removing noise through filtering.

[0239] Step 2:

[0240] The server inputs the denoised audio data into a speech recognition model and converts the auditory information into text data in real time. This text data is then sent to the subsequent translation process.

[0241] Step 3:

[0242] The server processes the converted text data through a multilingual translation model, enabling accurate translations into multiple specified languages. Each translation result is then ready for subtitle generation.

[0243] Step 4:

[0244] The server constructs timestamped subtitle data based on the translated text. This data is then formatted for use as subtitles.

[0245] Step 5:

[0246] The device analyzes the video data and suggests the optimal subtitle display style (color, font size, position, etc.) based on the scene's color tone and elements.

[0247] Step 6:

[0248] Users can review suggested subtitle styles through their device and manually customize the subtitle design as needed. Edits are immediately reflected in the system.

[0249] Step 7:

[0250] The server overlays user-subtitled or edited subtitles onto the video or live stream in real time, providing viewers with multilingual subtitles.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] Currently, providing multilingual video content in real time presents challenges in achieving high-quality translations and appropriate subtitles, resulting in delays and inaccuracies in providing information in multiple languages. Furthermore, subtitle formats may not suit viewers' visual preferences, potentially detracting from the viewing experience. To address these issues, a system is needed that instantly translates audio information and suggests the most suitable subtitle format based on the visual information.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing visual information and proposing a display format for the translated text information. This makes it possible to generate multilingual encoded subtitles in real time and provide them in a display format that suits visual preferences.

[0256] "Audio information" refers to sound signals transmitted through sound waves, including acoustic data such as human voices.

[0257] "Real-time" refers to a state where data processing and communication occur instantly, resulting in virtually no delay and immediate processing results.

[0258] "Textual information" refers to data that represents information transmitted through sound or sight as written characters in a language.

[0259] "Multiple languages" refers to a set of languages ​​with different linguistic systems, including various languages ​​used internationally.

[0260] Translation is the act of converting information expressed in one language into another language, thereby maintaining meaning while expressing equivalent content in a different linguistic system.

[0261] "Visual information" refers to data obtained through vision, including information such as videos and images.

[0262] "Display format" refers to the layout and style used when data or information is presented visually, and includes characteristics such as color, size, and position.

[0263] "Real-time composite display" refers to the process of instantly combining multiple data points and presenting them visually without delay.

[0264] To implement this invention, the system is configured as follows.

[0265] The server first acquires audio information from acoustic data collection devices or digital streaming services. This audio information is preprocessed using a noise reduction algorithm and then converted into text information using a speech recognition model (e.g., a speech recognition API). This conversion allows for the rapid and accurate conversion of information obtained from audio signals into text format.

[0266] Next, the server utilizes a multilingual translation model (e.g., a translation service API) to translate text information into multiple languages. The translated text information is managed appropriately in synchronization with the visual information. This process eliminates language barriers, allowing viewers to understand the content in their native language.

[0267] The terminal receives visual information from the video output device, analyzes that information, and automatically suggests the optimal display format. This includes settings such as color, font size, and placement, and is adjusted to provide the best possible display for the viewing environment and individual scenes.

[0268] Users can review and customize these suggestions via their devices. Specifically, they can adjust the subtitle display style to their preferences and needs by manipulating the device's user interface. This allows viewers to enjoy a visual experience tailored to their individual visual preferences.

[0269] Ultimately, the server integrates all settings and composites them into the live stream or video content in real time. This allows viewers to instantly understand information in multiple languages, enabling the provision of appropriate information to audiences with diverse cultural backgrounds.

[0270] As a concrete example, this system can be used in international online events to allow participants speaking different languages ​​to follow the discussion simultaneously in their respective languages. An example of a prompt would be, "Please translate the audio of this meeting into English, Spanish, and French in real time and display the subtitles." In response to this prompt, the server retrieves the audio information, translates it, and provides the information to each terminal in the appropriate display format.

[0271] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0272] Step 1:

[0273] The server acquires audio information from video streams and audio acquisition devices. The input is either an audio signal or streaming data, which the server records as an audio file. A noise reduction algorithm is applied to ensure the audio is clear. As a result, the server outputs audio data with the noise removed.

[0274] Step 2:

[0275] The server passes the denoised audio data to a speech recognition model for conversion to text. The input for this step is denoised audio data, and the server generates text data in real time using a speech recognition API. The output is text data converted from speech.

[0276] Step 3:

[0277] The server inputs the generated text data into a multilingual translation model and translates it into multiple languages specified by the user. The input for this step is the text data obtained from speech recognition, and the server outputs the text data translated into each language using the translation service API.

[0278] Step 4:

[0279] The terminal receives video data and analyzes visual information to assist in optimizing subtitle display. The input for this step is the video data, and the terminal uses a scene analysis algorithm to propose a subtitle display style suitable for the scene. The output is the setting of the proposed display style (color, size, position).

[0280] Step 5:

[0281] The user checks the proposed subtitle display style via the terminal interface and customizes it. The input is the proposed display style from the terminal, and the customized style settings are output by the user's operation. In this process, the user can adjust the font size, color, and modify the display position.

[0282] Step 6:

[0283] The server applies the subtitle display style customized by the user to the video and generates a real-time synthesized video. The input for this step is the translated text data and the customized display style, and the server integrates them to output the final video with subtitles. As a result, viewers can view multilingual subtitles in real time.

[0284] (Application Example 1)

[0285] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0286] In international content delivery services, there is a demand to provide subtitles corresponding to various languages in real time and to realize a visual display format that can be customized for each viewer. However, current methods have problems in translation accuracy and display flexibility, which may reduce the quality of the viewing experience. There is a need for a means to solve such problems and provide an efficient and high-quality multilingual subtitle service.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0288] In this invention, the server includes means for analyzing an audio signal in real time and converting it into document data, means for converting the document data into a plurality of natural languages, and means for analyzing a video signal and proposing a display format for subtitles. As a result, subtitles in various languages can be generated in real time, and a subtitle display that can be visually customized by the user becomes possible.

[0289] An "audio signal" is an electrical representation of sound information and is the object to be analyzed in real time and converted into text.

[0290] "Document data" is text information converted from an audio signal and is the basis for translation into multiple languages.

[0291] A "natural language" refers to words and languages that people use in daily life and is the object into which document data is translated.

[0292] A "video signal" is an electrical representation of visual information and is analyzed when proposing a display format for subtitles.

[0293] A "subtitle" is a visual display of the audio content within a video as text and is generated in real time.

[0294] A "display format" is an element that determines the appearance of subtitles and includes visual characteristics such as color, font size, and arrangement.

[0295] A "user" is an individual or group that can use the system to perform visual customizations.

[0296] A "server" is a computer device that performs audio signal analysis, translation, and subtitle generation, and forms the core of the system.

[0297] The system for implementing this invention mainly consists of a server and terminals. The central server analyzes speech signals in real time using Google Cloud Speech-to-Text and converts them into document data. Subsequently, this document data is translated into multiple natural languages ​​using the Google Cloud Translation API, and translated documents are obtained. The translated document data is used as subtitles, and data management is performed for visual customization.

[0298] The device provides an interface to the user through an application developed with React Native, suggesting subtitle display formats. Through this interface, the user can freely customize the subtitle's color, font size, and placement. Finally, the customized subtitles are visualized in real time during communication via the device.

[0299] A concrete example is that on international anime streaming platforms, this system allows viewers to easily access real-time subtitles in different languages. This enables multilingual subtitled streaming, allowing users from multiple countries to enjoy content simultaneously.

[0300] An example of a prompt using a generative AI model is, "Search for the anime title by voice, translate the audio from Japanese to English, and display the subtitles." This allows users to fully understand content even in languages ​​other than their native tongue.

[0301] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0302] Step 1:

[0303] The server obtains video and audio data in real time from the streaming source. It recognizes this data as an audio signal and applies a noise removal filter to generate clear audio data. As a result, high-quality input data for analysis is obtained.

[0304] Step 2:

[0305] The server analyzes the noise-removed audio signal using Google Cloud Speech-to-Text and converts it into document data. This document data is the text data necessary to accurately represent the audio content. This text data serves as the input for the translation process.

[0306] Step 3:

[0307] The server translates the document data generated in the previous step into a plurality of specified natural languages using the Google Cloud Translation API. Through this translation process, document data corresponding to each language is generated, preparing for the next subtitle generation step.

[0308] Step 4:

[0309] The server generates subtitle data based on the translated document data. This subtitle data contains time stamps and document information corresponding to each language, and it is necessary to be displayed in synchronization with the video.

[0310] Step 5:

[0311] The terminal analyzes this subtitle data and proposes a visually attractive display format to the user. Customization options for adjusting colors, font sizes, layouts, etc. are provided. Based on this information, the user determines the display format.

[0312] Step 6:

[0313] Users select and modify the subtitle display format using the interface on their device. The selected display format will be used in the next real-time display.

[0314] Step 7:

[0315] The device ultimately overlays the user-customized subtitle data onto the video, completing the visual output. This allows viewers to see subtitles in their chosen language in real time.

[0316] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0317] This invention provides a voice and video analysis system that combines an emotion engine to recognize the user's emotional state in real time and dynamically adjust the subtitle display style accordingly. This system has a multilingual subtitle generation function and provides visually and emotionally optimized content, taking into account the video content and the user's emotions.

[0318] The server processes video and live-stream audio data in real time and converts it into text data through a speech recognition model. This text information is translated within the server, generating subtitle data in multiple languages. Furthermore, the server uses an emotion engine to analyze the user's emotions from camera footage and audio acquired from the user's device. This analysis information influences the display format of the subtitles.

[0319] The device analyzes video data and suggests subtitle styles, such as color, font size, and position, based on the scene. Furthermore, the device dynamically adjusts the subtitle style in response to the user's emotions based on emotion information obtained from the emotion engine. For example, if the user is surprised, the font size of the subtitles can be increased to highlight them.

[0320] Users can review suggested subtitle styles through their device's interface and make manual adjustments. The emotion engine automatically provides optimal subtitles tailored to the user's emotions, without requiring any user changes to the default settings.

[0321] Furthermore, the server can generate visual images and illustrations based on emotion recognition and display them along with subtitles as needed, thereby further enhancing the impact of the video. This makes it possible to provide viewers with an even more engaging media experience.

[0322] This system enables the creation of highly responsive content that takes audience emotions into account, particularly in the entertainment and education sectors, significantly improving the user experience.

[0323] The following describes the processing flow.

[0324] Step 1:

[0325] The server acquires audio data from video and live streams, applies noise reduction filters, and prepares it for use in speech recognition models.

[0326] Step 2:

[0327] The server inputs the denoised audio into a speech recognition model and converts it into text data in real time. This data is used as the basis for subtitle generation.

[0328] Step 3:

[0329] The server passes the converted text data through a multilingual translation model and translates it into the specified multiple languages. This generates multilingual text that can be used as subtitles.

[0330] Step 4:

[0331] The server inputs camera footage and audio from the user's device into the emotion engine to analyze the user's emotional state. This information is used for dynamic adjustment of subtitle display.

[0332] Step 5:

[0333] Based on the analysis results from the emotion engine, the server generates instructions to optimize the display color, font size, and position of the subtitles.

[0334] Step 6:

[0335] The device analyzes video data and suggests subtitle display styles appropriate to the scene. It also adjusts the style based on the user's emotions, incorporating the results of the emotion engine.

[0336] Step 7:

[0337] Users can review and edit suggested subtitle styles on their device's interface. Even if the user does nothing, the system automatically adjusts the subtitle display to the optimal setting.

[0338] Step 8:

[0339] The server overlays user-reviewed or edited subtitles, along with optimization results from the emotion engine, onto the video or live stream in real time and provides them to viewers.

[0340] (Example 2)

[0341] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0342] In video and audio content, there is a need to dynamically adjust subtitle styles in real time according to emotions so that viewers can emotionally empathize with the content, transcending language and cultural differences. However, current systems struggle to efficiently and effectively achieve this requirement. In particular, integrating multilingual translation, emotion analysis, and visual adjustments is difficult, resulting in a less-than-optimal user experience.

[0343] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0344] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing the user's emotional state and dynamically adjusting the subtitle display style. This enables optimal subtitle display that responds to the viewer's emotions, achieving deep emotional resonance in international media content and dramatically improving the user experience.

[0345] "Audio information" refers to information that represents audio waveforms as digital data and is in a format that can be analyzed by speech recognition processes.

[0346] "Textual information" refers to data in text format obtained through the analysis of audio information.

[0347] "Translating into a language" is the process of converting written information from one language into another different language to convey its meaning.

[0348] "Visual information" refers to information that includes visual images and video data, and is used for the analysis of visual content.

[0349] "Subtitle display style" refers to the style and format of visually displayed text information, including elements such as color, font size, and position.

[0350] "Analyzing a user's emotional state" is the process of determining a user's emotions from their facial expressions, tone of voice, etc., and expressing that state quantitatively or qualitatively.

[0351] "Real-time display" refers to the process of immediately visualizing data after it has been generated or received.

[0352] "Visual information and shapes" refer to illustrations and graphic elements added to video information, which play a role in assisting the understanding of the content.

[0353] This invention is a system that analyzes audio and video data in real time and provides subtitles tailored to the user's emotions. It primarily functions through the interaction of a server, terminals, and users.

[0354] The server is responsible for converting speech information into text information using a speech recognition model. Specifically, a speech recognition API can be applied as the software. The converted text information is then translated into multiple languages ​​using multilingual translation software. In this process, a translation API is used to achieve efficient and accurate multilingual translation.

[0355] Furthermore, the server analyzes camera video and audio data transmitted from the user device through an emotion analysis engine to infer the user's emotional state. By utilizing an emotion recognition API, it becomes possible to determine emotions from the user's facial expressions and voice tone. This analysis result is used to dynamically adjust the display style of subtitles.

[0356] The device proposes subtitle styles based on translated subtitle data and sentiment analysis results sent from the server. Specifically, it uses video editing software to automatically generate a visual style optimized for the scene content and the user's emotional state. For example, in emotional scenes, highlighting using warm colors is suggested.

[0357] Users can review and manually adjust suggested display styles through the device interface. By customizing the style themselves, rather than relying on default settings, users can achieve a more personalized content presentation. Utilizing a generative AI model, users can input prompts such as "If surprised, change subtitles to red," enabling specific and intuitive customization.

[0358] This system has the potential to improve the user experience across various content sectors, including entertainment and education. Viewers can gain a new viewing experience where they can intuitively understand and emotionally empathize with content from different cultures and languages ​​in their own language.

[0359] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0360] Step 1:

[0361] The server receives audio data and converts it into text data using a speech recognition model. The input is audio data, and the output is the converted text data. By using a speech recognition API, the audio waveform is analyzed and text data is generated in real time.

[0362] Step 2:

[0363] The server inputs the character data obtained in step 1 into the translation module and performs multilingual translation. The input is character data, and the output is text data translated into multiple languages. A translation API is applied to perform fast and accurate translation.

[0364] Step 3:

[0365] The server uses an emotion analysis engine to evaluate the user's emotional state based on camera footage and audio sent from the user's device. The input is the user's video and audio data, and the output is the analysis result indicating their emotional state. It utilizes an emotion recognition API to analyze the user's facial expressions and voice and determine the corresponding emotions.

[0366] Step 4:

[0367] The terminal selects a subtitle style based on the translated text and sentiment analysis results received from the server. The input is the translated text and sentiment analysis results, and the output is the proposed subtitle style. Video editing software is then used to determine appropriate colors, font sizes, and other elements.

[0368] Step 5:

[0369] The user reviews the subtitle style displayed on the device and makes adjustments as needed. The input is the suggested style, and the output is the final adjusted style. The user can customize the subtitles, such as adjusting colors and changing fonts, using an intuitive interface.

[0370] Step 6:

[0371] The server receives the final subtitle style, integrates it into the video, and delivers it to viewers in real time. The input is the final style and video data, and the output is the video with subtitles. The goal is to enhance the viewer experience by providing them with emotionally optimized video content.

[0372] (Application Example 2)

[0373] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0374] Conventional subtitling systems simply convert audio to text and display subtitles, making it difficult to provide interactive content that takes into account the viewer's emotional state. Therefore, there is a need to provide a richer and more engaging viewing experience by analyzing the user's emotions in real time and automatically adjusting the subtitle display style accordingly.

[0375] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0376] In this invention, the server includes means for analyzing the user's emotional state obtained from audio and video information, means for dynamically adjusting the subtitle display style according to the user's emotional state, and means for creating relevant visual information using a generative AI model and suggesting its display along with subtitles as needed. This makes it possible to interactively adjust the content display style according to the viewer's emotions and provide a more engaging and personalized viewing experience.

[0377] "Acoustic information" refers to speech and other auditory data that can be converted into textual information through analysis.

[0378] "Textual information" refers to data in text format obtained by analyzing acoustic information, and is used for translation and subtitle generation.

[0379] "Visual information" refers to visual data, and analyzing this data is useful for determining the display format of subtitles and generating visual information.

[0380] "User emotional state" refers to data that indicates the user's emotional response, analyzed based on indicators obtained from video and audio information.

[0381] "Dynamically adjusting subtitle display style" refers to a function that changes the font, color, size, and placement of subtitles in real time according to the user's emotional state.

[0382] A "generative AI model" refers to artificial intelligence technology that generates new data or images based on input data, and is also used to create visual information.

[0383] "Visual information" refers to illustrations and images created using generative AI models, which are added to video content.

[0384] Subtitles are a way of displaying audio in video content as text, and they can also be used to support multiple languages.

[0385] The system realizing this invention connects the user's device (such as a smartphone or smart glasses) with a server and operates based on acoustic and video information. Acoustic information is collected through the device's microphone, and the server converts it into text information. Speech recognition technology is used for the conversion, and real-time processing is performed using Python and TensorFlow.

[0386] The server further translates the acquired text information into multiple languages ​​and automatically generates multilingual subtitles. In addition, it analyzes the user's emotional state from video and audio information using an emotion engine. OpenCV is used for video processing here. A function is incorporated to adjust the subtitle display style (font size, color, position, etc.) in real time according to the user's emotional state.

[0387] On the user's device, the generated subtitles are displayed in sync with the video. Additionally, a generative AI model is used to create related visual information, which is then displayed in conjunction with the subtitles. Specifically, generative models such as Stable Diffusion are utilized to dynamically generate illustrations and images that correspond to the user's emotional state.

[0388] For example, if the system detects a smile while a user is watching a comedy film, it can brighten the subtitles and add illustrations. This system allows subtitles to change according to the viewer's emotions while watching movies or videos, providing a personalized viewing experience.

[0389] An example of a prompt for the generation AI model would be, "Generate more colorful and humorous illustrations when the user is laughing."

[0390] This allows viewers to receive an optimized media experience that responds to their emotions.

[0391] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0392] Step 1:

[0393] The device collects acoustic information. This acoustic information is acquired by the device's microphone. When acoustic information is received, the device transmits it to the server in real time.

[0394] Step 2:

[0395] The server converts acoustic information into text information. The server takes acoustic information as input and outputs it as text information using speech recognition technology. Specifically, it uses TensorFlow to run a speech recognition model and outputs the resulting text data.

[0396] Step 3:

[0397] The server translates the text information. It takes the text information as input and uses a translation API to generate translated text for multiple languages. The output translated text is used for automatic subtitle generation.

[0398] Step 4:

[0399] The device analyzes the user's emotional state from video information. It analyzes the video information input to the device using OpenCV or similar tools, detecting the user's facial expressions and movements to determine their emotional state. The analysis results are sent to the server as emotional data.

[0400] Step 5:

[0401] The server adjusts the subtitle display style based on emotional state. It takes emotional data as input, dynamically sets the font size, color, and position of the subtitles based on this data, and outputs the adjusted subtitles.

[0402] Step 6:

[0403] The user's device generates visual information using a generative AI model. It takes emotional data and prompt text as input, and uses generative AI models such as Stable Diffusion to generate relevant visual information, which is then displayed alongside the video content. The output visual information serves as an additional element that complements the video experience.

[0404] Step 7:

[0405] Users can view subtitles that respond to their emotions and manually adjust the display style as needed. Users can use the settings screen as input to modify the display style. This allows for an even more personalized viewing experience.

[0406] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0407] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0408] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0409] [Third Embodiment]

[0410] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0411] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0412] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0413] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0414] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0415] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0416] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0417] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0418] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0419] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0420] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0421] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0422] This invention relates to a system that analyzes audio data in real time, translates it, and generates multilingual subtitles. The system aims to automate the entire process from audio input to subtitle display, enabling users to efficiently provide high-quality content.

[0423] The server first acquires audio data from video and live streams, performs noise reduction, and then converts it into text data using a speech recognition model. This enables real-time conversion from audio information to text information.

[0424] The acquired text data is translated by the server into the language specified by the user using a multilingual translation model. The translated text is then incorporated into a subtitle data structure within the server and managed along with information such as timestamps.

[0425] The device analyzes video data and suggests the optimal subtitle display style for each scene. This includes suggestions regarding subtitle color, font size, and position, supporting a visually appealing display for the user.

[0426] Users can view these suggestions via their device and customize them as needed. For example, users can adjust the font size to make subtitles more prominent in bright scenes.

[0427] Ultimately, the server overlays the edited subtitles onto the video or live stream in real time. This allows viewers to instantly see subtitles in different languages, enabling broadcasting that caters to a multinational audience.

[0428] This system is particularly useful in specific applications such as international conferences and online events targeting multinational audiences, enabling the provision of accurate multilingual information while simplifying user operation.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The server acquires audio data from video and live streams, and generates clear audio data by removing noise through filtering.

[0432] Step 2:

[0433] The server inputs the denoised audio data into a speech recognition model and converts the auditory information into text data in real time. This text data is then sent to the subsequent translation process.

[0434] Step 3:

[0435] The server processes the converted text data through a multilingual translation model, enabling accurate translations into multiple specified languages. Each translation result is then ready for subtitle generation.

[0436] Step 4:

[0437] The server constructs timestamped subtitle data based on the translated text. This data is then formatted for use as subtitles.

[0438] Step 5:

[0439] The device analyzes the video data and suggests the optimal subtitle display style (color, font size, position, etc.) based on the scene's color tone and elements.

[0440] Step 6:

[0441] Users can review suggested subtitle styles through their device and manually customize the subtitle design as needed. Edits are immediately reflected in the system.

[0442] Step 7:

[0443] The server overlays user-subtitled or edited subtitles onto the video or live stream in real time, providing viewers with multilingual subtitles.

[0444] (Example 1)

[0445] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] Currently, providing multilingual video content in real time presents challenges in achieving high-quality translations and appropriate subtitles, resulting in delays and inaccuracies in providing information in multiple languages. Furthermore, subtitle formats may not suit viewers' visual preferences, potentially detracting from the viewing experience. To address these issues, a system is needed that instantly translates audio information and suggests the most suitable subtitle format based on the visual information.

[0447] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0448] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing visual information and proposing a display format for the translated text information. This makes it possible to generate multilingual encoded subtitles in real time and provide them in a display format that suits visual preferences.

[0449] "Audio information" refers to sound signals transmitted through sound waves, including acoustic data such as human voices.

[0450] "Real-time" refers to a state where data processing and communication occur instantly, resulting in virtually no delay and immediate processing results.

[0451] "Textual information" refers to data that represents information transmitted through sound or sight as written characters in a language.

[0452] "Multiple languages" refers to a set of languages ​​with different linguistic systems, including various languages ​​used internationally.

[0453] Translation is the act of converting information expressed in one language into another language, thereby maintaining meaning while expressing equivalent content in a different linguistic system.

[0454] "Visual information" refers to data obtained through vision, including information such as videos and images.

[0455] "Display format" refers to the layout and style used when data or information is presented visually, and includes characteristics such as color, size, and position.

[0456] "Real-time composite display" refers to the process of instantly combining multiple data points and presenting them visually without delay.

[0457] To implement this invention, the system is configured as follows.

[0458] The server first acquires audio information from acoustic data collection devices or digital streaming services. This audio information is preprocessed using a noise reduction algorithm and then converted into text information using a speech recognition model (e.g., a speech recognition API). This conversion allows for the rapid and accurate conversion of information obtained from audio signals into text format.

[0459] Next, the server utilizes a multilingual translation model (e.g., a translation service API) to translate text information into multiple languages. The translated text information is managed appropriately in synchronization with the visual information. This process eliminates language barriers, allowing viewers to understand the content in their native language.

[0460] The terminal receives visual information from the video output device, analyzes that information, and automatically suggests the optimal display format. This includes settings such as color, font size, and placement, and is adjusted to provide the best possible display for the viewing environment and individual scenes.

[0461] Users can review and customize these suggestions via their devices. Specifically, they can adjust the subtitle display style to their preferences and needs by manipulating the device's user interface. This allows viewers to enjoy a visual experience tailored to their individual visual preferences.

[0462] Ultimately, the server integrates all settings and composites them into the live stream or video content in real time. This allows viewers to instantly understand information in multiple languages, enabling the provision of appropriate information to audiences with diverse cultural backgrounds.

[0463] As a concrete example, this system can be used in international online events to allow participants speaking different languages ​​to follow the discussion simultaneously in their respective languages. An example of a prompt would be, "Please translate the audio of this meeting into English, Spanish, and French in real time and display the subtitles." In response to this prompt, the server retrieves the audio information, translates it, and provides the information to each terminal in the appropriate display format.

[0464] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0465] Step 1:

[0466] The server acquires audio information from video streams and audio acquisition devices. The input is either an audio signal or streaming data, which the server records as an audio file. A noise reduction algorithm is applied to ensure the audio is clear. As a result, the server outputs audio data with the noise removed.

[0467] Step 2:

[0468] The server passes the denoised audio data to a speech recognition model for conversion to text. The input for this step is denoised audio data, and the server generates text data in real time using a speech recognition API. The output is text data converted from speech.

[0469] Step 3:

[0470] The server inputs the generated text data into a multilingual translation model and translates it into multiple languages ​​specified by the user. The input for this step is the text data obtained from speech recognition, and the server outputs the translated text data in each language using a translation service API.

[0471] Step 4:

[0472] The terminal receives video data and analyzes the visual information to help optimize subtitle display. The input for this step is video data, and the terminal uses a scene analysis algorithm to suggest a subtitle display style appropriate for the scene. The output is the settings (color, size, position) of the suggested display style.

[0473] Step 5:

[0474] The user reviews and customizes the suggested subtitle display style via the device's interface. The input is the suggested display style from the device, and the customized style settings are output based on the user's actions. During this process, the user can adjust font size and color, and correct the display position.

[0475] Step 6:

[0476] The server applies a user-customized subtitle display style to the video, generating a real-time composite video. The input for this step is translated text data and the customized display style, which the server integrates to output the final subtitled video. As a result, viewers can view multilingual subtitles in real time.

[0477] (Application Example 1)

[0478] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0479] International content distribution services require the real-time provision of subtitles in multiple languages ​​and the creation of customizable visual display formats for each viewer. However, current methods have limitations in translation accuracy and display flexibility, which can degrade the quality of the viewing experience. A solution is needed to address these issues and provide an efficient and high-quality multilingual subtitle service.

[0480] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0481] In this invention, the server includes means for analyzing an audio signal in real time and converting it into document data, means for converting the document data into multiple natural languages, and means for analyzing a video signal and proposing a subtitle display format. This enables the real-time generation of subtitles in various languages ​​and allows users to visually customize the subtitle display.

[0482] An "audio signal" is an electrical representation of sound information that is analyzed in real time and converted into text.

[0483] "Document data" refers to text information converted from audio signals, which serves as the basis for translation into multiple languages.

[0484] "Natural language" refers to the words and languages ​​that people use on a daily basis, and is the subject of document data translation.

[0485] A "video signal" is an electrical representation of visual information, and it is analyzed when proposing a subtitle display format.

[0486] Subtitles are a visual representation of the audio content within a video, and they are generated in real time.

[0487] "Display format" refers to the elements that determine the appearance of subtitles, and includes visual characteristics such as color, font size, and placement.

[0488] A "user" is an individual or group that can use the system to perform visual customizations.

[0489] A "server" is a computer device that performs audio signal analysis, translation, and subtitle generation, and forms the core of the system.

[0490] The system for implementing this invention mainly consists of a server and terminals. The central server analyzes speech signals in real time using Google Cloud Speech-to-Text and converts them into document data. Subsequently, this document data is translated into multiple natural languages ​​using the Google Cloud Translation API, and translated documents are obtained. The translated document data is used as subtitles, and data management is performed for visual customization.

[0491] The device provides an interface to the user through an application developed with React Native, suggesting subtitle display formats. Through this interface, the user can freely customize the subtitle's color, font size, and placement. Finally, the customized subtitles are visualized in real time during communication via the device.

[0492] A concrete example is that on international anime streaming platforms, this system allows viewers to easily access real-time subtitles in different languages. This enables multilingual subtitled streaming, allowing users from multiple countries to enjoy content simultaneously.

[0493] An example of a prompt using a generative AI model is, "Search for the anime title by voice, translate the audio from Japanese to English, and display the subtitles." This allows users to fully understand content even in languages ​​other than their native tongue.

[0494] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0495] Step 1:

[0496] The server acquires video and audio data from the streaming source in real time. It recognizes this data as an audio signal and applies a noise reduction filter to generate clear audio data. This provides high-quality input data for analysis.

[0497] Step 2:

[0498] The server analyzes the denoised audio signal using Google Cloud Speech-to-Text and converts it into document data. This document data is the text data necessary to accurately represent the content of the audio. This text data serves as the input for the translation process.

[0499] Step 3:

[0500] The server uses the Google Cloud Translation API to translate the document data generated in the previous step into the specified natural languages. This translation process generates document data corresponding to each language, preparing it for the next subtitle generation step.

[0501] Step 4:

[0502] The server generates subtitle data based on the translated document data. This subtitle data includes timestamps and document information specific to each language, and it needs to be displayed in sync with the video.

[0503] Step 5:

[0504] The device analyzes this subtitle data and suggests a visually appealing display format to the user. Customization options are provided to adjust colors, font size, and layout. Based on this information, the user decides on the display format.

[0505] Step 6:

[0506] Users select and modify the subtitle display format using the interface on their device. The selected display format will be used in the next real-time display.

[0507] Step 7:

[0508] The device ultimately overlays the user-customized subtitle data onto the video, completing the visual output. This allows viewers to see subtitles in their chosen language in real time.

[0509] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0510] This invention provides a voice and video analysis system that combines an emotion engine to recognize the user's emotional state in real time and dynamically adjust the subtitle display style accordingly. This system has a multilingual subtitle generation function and provides visually and emotionally optimized content, taking into account the video content and the user's emotions.

[0511] The server processes video and live-stream audio data in real time and converts it into text data through a speech recognition model. This text information is translated within the server, generating subtitle data in multiple languages. Furthermore, the server uses an emotion engine to analyze the user's emotions from camera footage and audio acquired from the user's device. This analysis information influences the display format of the subtitles.

[0512] The device analyzes video data and suggests subtitle styles, such as color, font size, and position, based on the scene. Furthermore, the device dynamically adjusts the subtitle style in response to the user's emotions based on emotion information obtained from the emotion engine. For example, if the user is surprised, the font size of the subtitles can be increased to highlight them.

[0513] Users can review suggested subtitle styles through their device's interface and make manual adjustments. The emotion engine automatically provides optimal subtitles tailored to the user's emotions, without requiring any user changes to the default settings.

[0514] Furthermore, the server can generate visual images and illustrations based on emotion recognition and display them along with subtitles as needed, thereby further enhancing the impact of the video. This makes it possible to provide viewers with an even more engaging media experience.

[0515] This system enables the creation of highly responsive content that takes audience emotions into account, particularly in the entertainment and education sectors, significantly improving the user experience.

[0516] The following describes the processing flow.

[0517] Step 1:

[0518] The server acquires audio data from video and live streams, applies noise reduction filters, and prepares it for use in speech recognition models.

[0519] Step 2:

[0520] The server inputs the denoised audio into a speech recognition model and converts it into text data in real time. This data is used as the basis for subtitle generation.

[0521] Step 3:

[0522] The server passes the converted text data through a multilingual translation model and translates it into the specified multiple languages. This generates multilingual text that can be used as subtitles.

[0523] Step 4:

[0524] The server inputs camera footage and audio from the user's device into the emotion engine to analyze the user's emotional state. This information is used for dynamic adjustment of subtitle display.

[0525] Step 5:

[0526] Based on the analysis results from the emotion engine, the server generates instructions to optimize the display color, font size, and position of the subtitles.

[0527] Step 6:

[0528] The device analyzes video data and suggests subtitle display styles appropriate to the scene. It also adjusts the style based on the user's emotions, incorporating the results of the emotion engine.

[0529] Step 7:

[0530] Users can review and edit suggested subtitle styles on their device's interface. Even if the user does nothing, the system automatically adjusts the subtitle display to the optimal setting.

[0531] Step 8:

[0532] The server overlays user-reviewed or edited subtitles, along with optimization results from the emotion engine, onto the video or live stream in real time and provides them to viewers.

[0533] (Example 2)

[0534] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0535] In video and audio content, there is a need to dynamically adjust subtitle styles in real time according to emotions so that viewers can emotionally empathize with the content, transcending language and cultural differences. However, current systems struggle to efficiently and effectively achieve this requirement. In particular, integrating multilingual translation, emotion analysis, and visual adjustments is difficult, resulting in a less-than-optimal user experience.

[0536] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0537] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing the user's emotional state and dynamically adjusting the subtitle display style. This enables optimal subtitle display that responds to the viewer's emotions, achieving deep emotional resonance in international media content and dramatically improving the user experience.

[0538] "Audio information" refers to information that represents audio waveforms as digital data and is in a format that can be analyzed by speech recognition processes.

[0539] "Textual information" refers to data in text format obtained through the analysis of audio information.

[0540] "Translating into a language" is the process of converting written information from one language into another different language to convey its meaning.

[0541] "Visual information" refers to information that includes visual images and video data, and is used for the analysis of visual content.

[0542] "Subtitle display style" refers to the style and format of visually displayed text information, including elements such as color, font size, and position.

[0543] "Analyzing a user's emotional state" is the process of determining a user's emotions from their facial expressions, tone of voice, etc., and expressing that state quantitatively or qualitatively.

[0544] "Real-time display" refers to the process of immediately visualizing data after it has been generated or received.

[0545] "Visual information and shapes" refer to illustrations and graphic elements added to video information, which play a role in assisting the understanding of the content.

[0546] This invention is a system that analyzes audio and video data in real time and provides subtitles tailored to the user's emotions. It primarily functions through the interaction of a server, terminals, and users.

[0547] The server is responsible for converting speech information into text information using a speech recognition model. Specifically, a speech recognition API can be applied as the software. The converted text information is then translated into multiple languages ​​using multilingual translation software. In this process, a translation API is used to achieve efficient and accurate multilingual translation.

[0548] Furthermore, the server analyzes camera video and audio data transmitted from the user device through an emotion analysis engine to infer the user's emotional state. By utilizing an emotion recognition API, it becomes possible to determine emotions from the user's facial expressions and voice tone. This analysis result is used to dynamically adjust the display style of subtitles.

[0549] The device proposes subtitle styles based on translated subtitle data and sentiment analysis results sent from the server. Specifically, it uses video editing software to automatically generate a visual style optimized for the scene content and the user's emotional state. For example, in emotional scenes, highlighting using warm colors is suggested.

[0550] Users can review and manually adjust suggested display styles through the device interface. By customizing the style themselves, rather than relying on default settings, users can achieve a more personalized content presentation. Utilizing a generative AI model, users can input prompts such as "If surprised, change subtitles to red," enabling specific and intuitive customization.

[0551] This system has the potential to improve the user experience across various content sectors, including entertainment and education. Viewers can gain a new viewing experience where they can intuitively understand and emotionally empathize with content from different cultures and languages ​​in their own language.

[0552] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0553] Step 1:

[0554] The server receives audio data and converts it into text data using a speech recognition model. The input is audio data, and the output is the converted text data. By using a speech recognition API, the audio waveform is analyzed and text data is generated in real time.

[0555] Step 2:

[0556] The server inputs the character data obtained in step 1 into the translation module and performs multilingual translation. The input is character data, and the output is text data translated into multiple languages. A translation API is applied to perform fast and accurate translation.

[0557] Step 3:

[0558] The server uses an emotion analysis engine to evaluate the user's emotional state based on camera footage and audio sent from the user's device. The input is the user's video and audio data, and the output is the analysis result indicating their emotional state. It utilizes an emotion recognition API to analyze the user's facial expressions and voice and determine the corresponding emotions.

[0559] Step 4:

[0560] The terminal selects a subtitle style based on the translated text and sentiment analysis results received from the server. The input is the translated text and sentiment analysis results, and the output is the proposed subtitle style. Video editing software is then used to determine appropriate colors, font sizes, and other elements.

[0561] Step 5:

[0562] The user reviews the subtitle style displayed on the device and makes adjustments as needed. The input is the suggested style, and the output is the final adjusted style. The user can customize the subtitles, such as adjusting colors and changing fonts, using an intuitive interface.

[0563] Step 6:

[0564] The server receives the final subtitle style, integrates it into the video, and delivers it to viewers in real time. The input is the final style and video data, and the output is the video with subtitles. The goal is to enhance the viewer experience by providing them with emotionally optimized video content.

[0565] (Application Example 2)

[0566] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0567] Conventional subtitling systems simply convert audio to text and display subtitles, making it difficult to provide interactive content that takes into account the viewer's emotional state. Therefore, there is a need to provide a richer and more engaging viewing experience by analyzing the user's emotions in real time and automatically adjusting the subtitle display style accordingly.

[0568] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0569] In this invention, the server includes means for analyzing the user's emotional state obtained from audio and video information, means for dynamically adjusting the subtitle display style according to the user's emotional state, and means for creating relevant visual information using a generative AI model and suggesting its display along with subtitles as needed. This makes it possible to interactively adjust the content display style according to the viewer's emotions and provide a more engaging and personalized viewing experience.

[0570] "Acoustic information" refers to speech and other auditory data that can be converted into textual information through analysis.

[0571] "Textual information" refers to data in text format obtained by analyzing acoustic information, and is used for translation and subtitle generation.

[0572] "Visual information" refers to visual data, and analyzing this data is useful for determining the display format of subtitles and generating visual information.

[0573] "User emotional state" refers to data that indicates the user's emotional response, analyzed based on indicators obtained from video and audio information.

[0574] "Dynamically adjusting subtitle display style" refers to a function that changes the font, color, size, and placement of subtitles in real time according to the user's emotional state.

[0575] A "generative AI model" refers to artificial intelligence technology that generates new data or images based on input data, and is also used to create visual information.

[0576] "Visual information" refers to illustrations and images created using generative AI models, which are added to video content.

[0577] Subtitles are a way of displaying audio in video content as text, and they can also be used to support multiple languages.

[0578] The system realizing this invention connects the user's device (such as a smartphone or smart glasses) with a server and operates based on acoustic and video information. Acoustic information is collected through the device's microphone, and the server converts it into text information. Speech recognition technology is used for the conversion, and real-time processing is performed using Python and TensorFlow.

[0579] The server further translates the acquired text information into multiple languages ​​and automatically generates multilingual subtitles. In addition, it analyzes the user's emotional state from video and audio information using an emotion engine. OpenCV is used for video processing here. A function is incorporated to adjust the subtitle display style (font size, color, position, etc.) in real time according to the user's emotional state.

[0580] On the user's device, the generated subtitles are displayed in sync with the video. Additionally, a generative AI model is used to create related visual information, which is then displayed in conjunction with the subtitles. Specifically, generative models such as Stable Diffusion are utilized to dynamically generate illustrations and images that correspond to the user's emotional state.

[0581] For example, if the system detects a smile while a user is watching a comedy film, it can brighten the subtitles and add illustrations. This system allows subtitles to change according to the viewer's emotions while watching movies or videos, providing a personalized viewing experience.

[0582] An example of a prompt for the generation AI model would be, "Generate more colorful and humorous illustrations when the user is laughing."

[0583] This allows viewers to receive an optimized media experience that responds to their emotions.

[0584] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0585] Step 1:

[0586] The device collects acoustic information. This acoustic information is acquired by the device's microphone. When acoustic information is received, the device transmits it to the server in real time.

[0587] Step 2:

[0588] The server converts acoustic information into text information. The server takes acoustic information as input and outputs it as text information using speech recognition technology. Specifically, it uses TensorFlow to run a speech recognition model and outputs the resulting text data.

[0589] Step 3:

[0590] The server translates the text information. It takes the text information as input and uses a translation API to generate translated text for multiple languages. The output translated text is used for automatic subtitle generation.

[0591] Step 4:

[0592] The device analyzes the user's emotional state from video information. It analyzes the video information input to the device using OpenCV or similar tools, detecting the user's facial expressions and movements to determine their emotional state. The analysis results are sent to the server as emotional data.

[0593] Step 5:

[0594] The server adjusts the subtitle display style based on emotional state. It takes emotional data as input, dynamically sets the font size, color, and position of the subtitles based on this data, and outputs the adjusted subtitles.

[0595] Step 6:

[0596] The user's device generates visual information using a generative AI model. It takes emotional data and prompt text as input, and uses generative AI models such as Stable Diffusion to generate relevant visual information, which is then displayed alongside the video content. The output visual information serves as an additional element that complements the video experience.

[0597] Step 7:

[0598] Users can view subtitles that respond to their emotions and manually adjust the display style as needed. Users can use the settings screen as input to modify the display style. This allows for an even more personalized viewing experience.

[0599] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0600] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0601] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0602] [Fourth Embodiment]

[0603] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0604] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0605] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0606] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0607] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0608] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0609] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0610] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0611] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0612] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0613] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0614] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0615] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0616] This invention relates to a system that analyzes audio data in real time, translates it, and generates multilingual subtitles. The system aims to automate the entire process from audio input to subtitle display, enabling users to efficiently provide high-quality content.

[0617] The server first acquires audio data from video and live streams, performs noise reduction, and then converts it into text data using a speech recognition model. This enables real-time conversion from audio information to text information.

[0618] The acquired text data is translated by the server into the language specified by the user using a multilingual translation model. The translated text is then incorporated into a subtitle data structure within the server and managed along with information such as timestamps.

[0619] The device analyzes video data and suggests the optimal subtitle display style for each scene. This includes suggestions regarding subtitle color, font size, and position, supporting a visually appealing display for the user.

[0620] Users can view these suggestions via their device and customize them as needed. For example, users can adjust the font size to make subtitles more prominent in bright scenes.

[0621] Ultimately, the server overlays the edited subtitles onto the video or live stream in real time. This allows viewers to instantly see subtitles in different languages, enabling broadcasting that caters to a multinational audience.

[0622] This system is particularly useful in specific applications such as international conferences and online events targeting multinational audiences, enabling the provision of accurate multilingual information while simplifying user operation.

[0623] The following describes the processing flow.

[0624] Step 1:

[0625] The server acquires audio data from video and live streams, and generates clear audio data by removing noise through filtering.

[0626] Step 2:

[0627] The server inputs the denoised audio data into a speech recognition model and converts the auditory information into text data in real time. This text data is then sent to the subsequent translation process.

[0628] Step 3:

[0629] The server processes the converted text data through a multilingual translation model, enabling accurate translations into multiple specified languages. Each translation result is then ready for subtitle generation.

[0630] Step 4:

[0631] The server constructs timestamped subtitle data based on the translated text. This data is then formatted for use as subtitles.

[0632] Step 5:

[0633] The device analyzes the video data and suggests the optimal subtitle display style (color, font size, position, etc.) based on the scene's color tone and elements.

[0634] Step 6:

[0635] Users can review suggested subtitle styles through their device and manually customize the subtitle design as needed. Edits are immediately reflected in the system.

[0636] Step 7:

[0637] The server overlays user-subtitled or edited subtitles onto the video or live stream in real time, providing viewers with multilingual subtitles.

[0638] (Example 1)

[0639] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0640] Currently, providing multilingual video content in real time presents challenges in achieving high-quality translations and appropriate subtitles, resulting in delays and inaccuracies in providing information in multiple languages. Furthermore, subtitle formats may not suit viewers' visual preferences, potentially detracting from the viewing experience. To address these issues, a system is needed that instantly translates audio information and suggests the most suitable subtitle format based on the visual information.

[0641] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0642] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing visual information and proposing a display format for the translated text information. This makes it possible to generate multilingual encoded subtitles in real time and provide them in a display format that suits visual preferences.

[0643] "Audio information" refers to sound signals transmitted through sound waves, including acoustic data such as human voices.

[0644] "Real-time" refers to a state where data processing and communication occur instantly, resulting in virtually no delay and immediate processing results.

[0645] "Textual information" refers to data that represents information transmitted through sound or sight as written characters in a language.

[0646] "Multiple languages" refers to a set of languages ​​with different linguistic systems, including various languages ​​used internationally.

[0647] Translation is the act of converting information expressed in one language into another language, thereby maintaining meaning while expressing equivalent content in a different linguistic system.

[0648] "Visual information" refers to data obtained through vision, including information such as videos and images.

[0649] "Display format" refers to the layout and style used when data or information is presented visually, and includes characteristics such as color, size, and position.

[0650] "Real-time composite display" refers to the process of instantly combining multiple data points and presenting them visually without delay.

[0651] To implement this invention, the system is configured as follows.

[0652] The server first acquires audio information from acoustic data collection devices or digital streaming services. This audio information is preprocessed using a noise reduction algorithm and then converted into text information using a speech recognition model (e.g., a speech recognition API). This conversion allows for the rapid and accurate conversion of information obtained from audio signals into text format.

[0653] Next, the server utilizes a multilingual translation model (e.g., a translation service API) to translate text information into multiple languages. The translated text information is managed appropriately in synchronization with the visual information. This process eliminates language barriers, allowing viewers to understand the content in their native language.

[0654] The terminal receives visual information from the video output device, analyzes that information, and automatically suggests the optimal display format. This includes settings such as color, font size, and placement, and is adjusted to provide the best possible display for the viewing environment and individual scenes.

[0655] Users can review and customize these suggestions via their devices. Specifically, they can adjust the subtitle display style to their preferences and needs by manipulating the device's user interface. This allows viewers to enjoy a visual experience tailored to their individual visual preferences.

[0656] Ultimately, the server integrates all settings and composites them into the live stream or video content in real time. This allows viewers to instantly understand information in multiple languages, enabling the provision of appropriate information to audiences with diverse cultural backgrounds.

[0657] As a concrete example, this system can be used in international online events to allow participants speaking different languages ​​to follow the discussion simultaneously in their respective languages. An example of a prompt would be, "Please translate the audio of this meeting into English, Spanish, and French in real time and display the subtitles." In response to this prompt, the server retrieves the audio information, translates it, and provides the information to each terminal in the appropriate display format.

[0658] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0659] Step 1:

[0660] The server acquires audio information from video streams and audio acquisition devices. The input is either an audio signal or streaming data, which the server records as an audio file. A noise reduction algorithm is applied to ensure the audio is clear. As a result, the server outputs audio data with the noise removed.

[0661] Step 2:

[0662] The server passes the denoised audio data to a speech recognition model for conversion to text. The input for this step is denoised audio data, and the server generates text data in real time using a speech recognition API. The output is text data converted from speech.

[0663] Step 3:

[0664] The server inputs the generated text data into a multilingual translation model and translates it into multiple languages ​​specified by the user. The input for this step is the text data obtained from speech recognition, and the server outputs the translated text data in each language using a translation service API.

[0665] Step 4:

[0666] The terminal receives video data and analyzes the visual information to help optimize subtitle display. The input for this step is video data, and the terminal uses a scene analysis algorithm to suggest a subtitle display style appropriate for the scene. The output is the settings (color, size, position) of the suggested display style.

[0667] Step 5:

[0668] The user reviews and customizes the suggested subtitle display style via the device's interface. The input is the suggested display style from the device, and the customized style settings are output based on the user's actions. During this process, the user can adjust font size and color, and correct the display position.

[0669] Step 6:

[0670] The server applies a user-customized subtitle display style to the video, generating a composite video in real time. The input for this step is translated text data and the customized display style, which the server integrates to output the final subtitled video. As a result, viewers can view multilingual subtitles in real time.

[0671] (Application Example 1)

[0672] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0673] International content distribution services require the real-time provision of subtitles in multiple languages ​​and the creation of customizable visual display formats for each viewer. However, current methods have limitations in translation accuracy and display flexibility, which can degrade the quality of the viewing experience. A solution is needed to address these issues and provide an efficient and high-quality multilingual subtitle service.

[0674] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0675] In this invention, the server includes means for analyzing an audio signal in real time and converting it into document data, means for converting the document data into multiple natural languages, and means for analyzing a video signal and proposing a subtitle display format. This enables the real-time generation of subtitles in various languages ​​and allows users to visually customize the subtitle display.

[0676] An "audio signal" is an electrical representation of sound information that is analyzed in real time and converted into text.

[0677] "Document data" refers to text information converted from audio signals, which serves as the basis for translation into multiple languages.

[0678] "Natural language" refers to the words and languages ​​that people use on a daily basis, and is the subject of document data translation.

[0679] A "video signal" is an electrical representation of visual information, and it is analyzed when proposing a subtitle display format.

[0680] Subtitles are a visual representation of the audio content within a video, and they are generated in real time.

[0681] "Display format" refers to the elements that determine the appearance of subtitles, and includes visual characteristics such as color, font size, and placement.

[0682] A "user" is an individual or group that can use the system to perform visual customizations.

[0683] A "server" is a computer device that performs audio signal analysis, translation, and subtitle generation, and forms the core of the system.

[0684] The system for implementing this invention mainly consists of a server and terminals. The central server analyzes speech signals in real time using Google Cloud Speech-to-Text and converts them into document data. Subsequently, this document data is translated into multiple natural languages ​​using the Google Cloud Translation API, and translated documents are obtained. The translated document data is used as subtitles, and data management is performed for visual customization.

[0685] The device provides an interface to the user through an application developed with React Native, suggesting subtitle display formats. Through this interface, the user can freely customize the subtitle's color, font size, and placement. Finally, the customized subtitles are visualized in real time during communication via the device.

[0686] A concrete example is that on international anime streaming platforms, this system allows viewers to easily access real-time subtitles in different languages. This enables multilingual subtitled streaming, allowing users from multiple countries to enjoy content simultaneously.

[0687] An example of a prompt using a generative AI model is, "Search for the anime title by voice, translate the audio from Japanese to English, and display the subtitles." This allows users to fully understand content even in languages ​​other than their native tongue.

[0688] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0689] Step 1:

[0690] The server acquires video and audio data from the streaming source in real time. It recognizes this data as an audio signal and applies a noise reduction filter to generate clear audio data. This provides high-quality input data for analysis.

[0691] Step 2:

[0692] The server analyzes the denoised audio signal using Google Cloud Speech-to-Text and converts it into document data. This document data is the text data necessary to accurately represent the content of the audio. This text data serves as the input for the translation process.

[0693] Step 3:

[0694] The server uses the Google Cloud Translation API to translate the document data generated in the previous step into the specified natural languages. This translation process generates document data corresponding to each language, preparing it for the next subtitle generation step.

[0695] Step 4:

[0696] The server generates subtitle data based on the translated document data. This subtitle data includes timestamps and document information specific to each language, and it needs to be displayed in sync with the video.

[0697] Step 5:

[0698] The device analyzes this subtitle data and suggests a visually appealing display format to the user. Customization options are provided to adjust colors, font size, and layout. Based on this information, the user decides on the display format.

[0699] Step 6:

[0700] Users select and modify the subtitle display format using the interface on their device. The selected display format will be used in the next real-time display.

[0701] Step 7:

[0702] The device ultimately overlays the user-customized subtitle data onto the video, completing the visual output. This allows viewers to see subtitles in their chosen language in real time.

[0703] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0704] This invention provides a voice and video analysis system that combines an emotion engine to recognize the user's emotional state in real time and dynamically adjust the subtitle display style accordingly. This system has a multilingual subtitle generation function and provides visually and emotionally optimized content, taking into account the video content and the user's emotions.

[0705] The server processes video and live-stream audio data in real time and converts it into text data through a speech recognition model. This text information is translated within the server, generating subtitle data in multiple languages. Furthermore, the server uses an emotion engine to analyze the user's emotions from camera footage and audio acquired from the user's device. This analysis information influences the display format of the subtitles.

[0706] The device analyzes video data and suggests subtitle styles, such as color, font size, and position, based on the scene. Furthermore, the device dynamically adjusts the subtitle style in response to the user's emotions based on emotion information obtained from the emotion engine. For example, if the user is surprised, the font size of the subtitles can be increased to highlight them.

[0707] Users can review suggested subtitle styles through their device's interface and make manual adjustments. The emotion engine automatically provides optimal subtitles tailored to the user's emotions, without requiring any user changes to the default settings.

[0708] Furthermore, the server can generate visual images and illustrations based on emotion recognition and display them along with subtitles as needed, thereby further enhancing the impact of the video. This makes it possible to provide viewers with an even more engaging media experience.

[0709] This system enables the creation of highly responsive content that takes audience emotions into account, particularly in the entertainment and education sectors, significantly improving the user experience.

[0710] The following describes the processing flow.

[0711] Step 1:

[0712] The server acquires audio data from video and live streams, applies noise reduction filters, and prepares it for use in speech recognition models.

[0713] Step 2:

[0714] The server inputs the denoised audio into a speech recognition model and converts it into text data in real time. This data is used as the basis for subtitle generation.

[0715] Step 3:

[0716] The server passes the converted text data through a multilingual translation model and translates it into the specified multiple languages. This generates multilingual text that can be used as subtitles.

[0717] Step 4:

[0718] The server inputs camera footage and audio from the user's device into the emotion engine to analyze the user's emotional state. This information is used for dynamic adjustment of subtitle display.

[0719] Step 5:

[0720] Based on the analysis results from the emotion engine, the server generates instructions to optimize the display color, font size, and position of the subtitles.

[0721] Step 6:

[0722] The device analyzes video data and suggests subtitle display styles appropriate to the scene. It also adjusts the style based on the user's emotions, incorporating the results of the emotion engine.

[0723] Step 7:

[0724] Users can review and edit suggested subtitle styles on their device's interface. Even if the user does nothing, the system automatically adjusts the subtitle display to the optimal setting.

[0725] Step 8:

[0726] The server overlays user-reviewed or edited subtitles, along with optimization results from the emotion engine, onto the video or live stream in real time and provides them to viewers.

[0727] (Example 2)

[0728] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0729] In video and audio content, there is a need to dynamically adjust subtitle styles in real time according to emotions so that viewers can emotionally empathize with the content, transcending language and cultural differences. However, current systems struggle to efficiently and effectively achieve this requirement. In particular, integrating multilingual translation, emotion analysis, and visual adjustments is difficult, resulting in a less-than-optimal user experience.

[0730] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0731] In this invention, the server includes means for analyzing audio information in real time and converting it into text information, means for translating the text information into multiple languages, and means for analyzing the user's emotional state and dynamically adjusting the subtitle display style. This enables optimal subtitle display that responds to the viewer's emotions, achieving deep emotional resonance in international media content and dramatically improving the user experience.

[0732] "Audio information" refers to information that represents audio waveforms as digital data and is in a format that can be analyzed by speech recognition processes.

[0733] "Textual information" refers to data in text format obtained through the analysis of audio information.

[0734] "Translating into a language" is the process of converting written information from one language into another different language to convey its meaning.

[0735] "Visual information" refers to information that includes visual images and video data, and is used for the analysis of visual content.

[0736] "Subtitle display style" refers to the style and format of visually displayed text information, including elements such as color, font size, and position.

[0737] "Analyzing a user's emotional state" is the process of determining a user's emotions from their facial expressions, tone of voice, etc., and expressing that state quantitatively or qualitatively.

[0738] "Real-time display" refers to the process of immediately visualizing data after it has been generated or received.

[0739] "Visual information and shapes" refer to illustrations and graphic elements added to video information, which play a role in assisting the understanding of the content.

[0740] This invention is a system that analyzes audio and video data in real time and provides subtitles tailored to the user's emotions. It primarily functions through the interaction of a server, terminals, and users.

[0741] The server is responsible for converting speech information into text information using a speech recognition model. Specifically, a speech recognition API can be applied as the software. The converted text information is then translated into multiple languages ​​using multilingual translation software. In this process, a translation API is used to achieve efficient and accurate multilingual translation.

[0742] Furthermore, the server analyzes camera video and audio data transmitted from the user device through an emotion analysis engine to infer the user's emotional state. By utilizing an emotion recognition API, it becomes possible to determine emotions from the user's facial expressions and voice tone. This analysis result is used to dynamically adjust the display style of subtitles.

[0743] The device proposes subtitle styles based on translated subtitle data and sentiment analysis results sent from the server. Specifically, it uses video editing software to automatically generate a visual style optimized for the scene content and the user's emotional state. For example, in emotional scenes, highlighting using warm colors is suggested.

[0744] Users can review and manually adjust suggested display styles through the device interface. By customizing the style themselves, rather than relying on default settings, users can achieve a more personalized content presentation. Utilizing a generative AI model, users can input prompts such as "If surprised, change subtitles to red," enabling specific and intuitive customization.

[0745] This system has the potential to improve the user experience across various content sectors, including entertainment and education. Viewers can gain a new viewing experience where they can intuitively understand and emotionally empathize with content from different cultures and languages ​​in their own language.

[0746] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0747] Step 1:

[0748] The server receives audio data and converts it into text data using a speech recognition model. The input is audio data, and the output is the converted text data. By using a speech recognition API, the audio waveform is analyzed and text data is generated in real time.

[0749] Step 2:

[0750] The server inputs the character data obtained in step 1 into the translation module and performs multilingual translation. The input is character data, and the output is text data translated into multiple languages. A translation API is applied to perform fast and accurate translation.

[0751] Step 3:

[0752] The server uses an emotion analysis engine to evaluate the user's emotional state based on camera footage and audio sent from the user's device. The input is the user's video and audio data, and the output is the analysis result indicating their emotional state. It utilizes an emotion recognition API to analyze the user's facial expressions and voice and determine the corresponding emotions.

[0753] Step 4:

[0754] The terminal selects a subtitle style based on the translated text and sentiment analysis results received from the server. The input is the translated text and sentiment analysis results, and the output is the proposed subtitle style. Video editing software is then used to determine appropriate colors, font sizes, and other elements.

[0755] Step 5:

[0756] The user reviews the subtitle style displayed on the device and makes adjustments as needed. The input is the suggested style, and the output is the final adjusted style. The user can customize the subtitles, such as adjusting colors and changing fonts, using an intuitive interface.

[0757] Step 6:

[0758] The server receives the final subtitle style, integrates it into the video, and delivers it to viewers in real time. The input is the final style and video data, and the output is the video with subtitles. The goal is to enhance the viewer experience by providing them with emotionally optimized video content.

[0759] (Application Example 2)

[0760] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0761] Conventional subtitling systems simply convert audio to text and display subtitles, making it difficult to provide interactive content that takes into account the viewer's emotional state. Therefore, there is a need to provide a richer and more engaging viewing experience by analyzing the user's emotions in real time and automatically adjusting the subtitle display style accordingly.

[0762] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0763] In this invention, the server includes means for analyzing the user's emotional state obtained from audio and video information, means for dynamically adjusting the subtitle display style according to the user's emotional state, and means for creating relevant visual information using a generative AI model and suggesting its display along with subtitles as needed. This makes it possible to interactively adjust the content display style according to the viewer's emotions and provide a more engaging and personalized viewing experience.

[0764] "Acoustic information" refers to speech and other auditory data that can be converted into textual information through analysis.

[0765] "Textual information" refers to data in text format obtained by analyzing acoustic information, and is used for translation and subtitle generation.

[0766] "Visual information" refers to visual data, and analyzing this data is useful for determining the display format of subtitles and generating visual information.

[0767] "User emotional state" refers to data that indicates the user's emotional response, analyzed based on indicators obtained from video and audio information.

[0768] "Dynamically adjusting subtitle display style" refers to a function that changes the font, color, size, and placement of subtitles in real time according to the user's emotional state.

[0769] A "generative AI model" refers to artificial intelligence technology that generates new data or images based on input data, and is also used to create visual information.

[0770] "Visual information" refers to illustrations and images created using generative AI models, which are added to video content.

[0771] Subtitles are a way of displaying audio in video content as text, and they can also be used to support multiple languages.

[0772] The system realizing this invention connects the user's device (such as a smartphone or smart glasses) with a server and operates based on acoustic and video information. Acoustic information is collected through the device's microphone, and the server converts it into text information. Speech recognition technology is used for the conversion, and real-time processing is performed using Python and TensorFlow.

[0773] The server further translates the acquired text information into multiple languages ​​and automatically generates multilingual subtitles. In addition, it analyzes the user's emotional state from video and audio information using an emotion engine. OpenCV is used for video processing here. A function is incorporated to adjust the subtitle display style (font size, color, position, etc.) in real time according to the user's emotional state.

[0774] On the user's device, the generated subtitles are displayed in sync with the video. Additionally, a generative AI model is used to create related visual information, which is then displayed in conjunction with the subtitles. Specifically, generative models such as Stable Diffusion are utilized to dynamically generate illustrations and images that correspond to the user's emotional state.

[0775] For example, if the system detects a smile while a user is watching a comedy film, it can brighten the subtitles and add illustrations. This system allows subtitles to change according to the viewer's emotions while watching movies or videos, providing a personalized viewing experience.

[0776] An example of a prompt for the generation AI model would be, "Generate more colorful and humorous illustrations when the user is laughing."

[0777] This allows viewers to receive an optimized media experience that responds to their emotions.

[0778] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0779] Step 1:

[0780] The device collects acoustic information. This acoustic information is acquired by the device's microphone. When acoustic information is received, the device transmits it to the server in real time.

[0781] Step 2:

[0782] The server converts acoustic information into text information. The server takes acoustic information as input and outputs it as text information using speech recognition technology. Specifically, it uses TensorFlow to run a speech recognition model and outputs the resulting text data.

[0783] Step 3:

[0784] The server translates the text information. It takes the text information as input and uses a translation API to generate translated text for multiple languages. The output translated text is used for automatic subtitle generation.

[0785] Step 4:

[0786] The device analyzes the user's emotional state from video information. It analyzes the video information input to the device using OpenCV or similar tools, detecting the user's facial expressions and movements to determine their emotional state. The analysis results are sent to the server as emotional data.

[0787] Step 5:

[0788] The server adjusts the subtitle display style based on emotional state. It takes emotional data as input, dynamically sets the font size, color, and position of the subtitles based on this data, and outputs the adjusted subtitles.

[0789] Step 6:

[0790] The user's device generates visual information using a generative AI model. It takes emotional data and prompt text as input, and uses generative AI models such as Stable Diffusion to generate relevant visual information, which is then displayed alongside the video content. The output visual information serves as an additional element that complements the video experience.

[0791] Step 7:

[0792] Users can view subtitles that respond to their emotions and manually adjust the display style as needed. Users can use the settings screen as input to modify the display style. This allows for an even more personalized viewing experience.

[0793] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0794] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0795] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0796] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0797] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0798] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0799] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0800] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0801] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0802] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0803] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0804] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0805] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0806] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0807] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0808] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0809] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0810] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0811] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0812] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0813] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0814] The following is further disclosed regarding the embodiments described above.

[0815] (Claim 1)

[0816] A means of analyzing audio data in real time and converting it into text data,

[0817] A means for translating the aforementioned text data into multiple languages,

[0818] A method for analyzing video data and proposing subtitle display styles,

[0819] A means for automatically generating subtitles using the translated texts of the aforementioned multiple languages,

[0820] A means to allow the user to edit the display style of the aforementioned subtitles,

[0821] A means of displaying the aforementioned subtitles in real time during a live broadcast,

[0822] A system that includes this.

[0823] (Claim 2)

[0824] The system according to claim 1, wherein the display style of the subtitles includes color, font size, and position.

[0825] (Claim 3)

[0826] The system according to claim 1, which generates and proposes the insertion of related image diagrams and illustrations based on the analysis of the aforementioned video data.

[0827] "Example 1"

[0828] (Claim 1)

[0829] A means of analyzing audio information in real time and converting it into text information,

[0830] A means for translating the aforementioned textual information into multiple languages,

[0831] A means of proposing a display format for translated text information by analyzing visual information,

[0832] A means for automatically generating subtitles using translated text information in the aforementioned multiple languages,

[0833] Means that allow users to modify the display format of the subtitles,

[0834] A means for displaying the aforementioned subtitles in real time during a live stream,

[0835] A system that includes this.

[0836] (Claim 2)

[0837] The system according to claim 1, wherein the display format of the subtitles includes color, size, and position.

[0838] (Claim 3)

[0839] The system according to claim 1, which generates and proposes the insertion of relevant visual materials and images based on the analysis of the aforementioned visual information.

[0840] "Application Example 1"

[0841] (Claim 1)

[0842] A means for analyzing audio signals in real time and converting them into document data,

[0843] Means for converting the aforementioned document data into multiple natural languages,

[0844] A method for analyzing video signals and proposing subtitle display formats,

[0845] A means for automatically generating subtitles using the aforementioned multiple natural language conversion documents,

[0846] Means that allow users to modify the display format of the subtitles,

[0847] A means for visualizing the aforementioned subtitles in real time during communication distribution,

[0848] Means for providing a user interface for visual customization to the device for visualizing the subtitles,

[0849] A system that includes this.

[0850] (Claim 2)

[0851] The system according to claim 1, wherein the display format of the subtitles includes color, font size, and placement.

[0852] (Claim 3)

[0853] The system according to claim 1, which generates relevant visual information and diagrams and proposes their arrangement based on the analysis of the aforementioned video signal.

[0854] "Example 2 of combining an emotion engine"

[0855] (Claim 1)

[0856] A means of analyzing audio information in real time and converting it into text information,

[0857] A means for translating the aforementioned textual information into multiple languages,

[0858] A means of analyzing video information and proposing a subtitle display format,

[0859] A means for automatically generating subtitles using translated characters from the aforementioned multiple languages,

[0860] Means for allowing the user to edit the display format of the subtitles,

[0861] A means for analyzing the user's emotional state and dynamically adjusting the display format of the subtitles,

[0862] A means of displaying the aforementioned subtitles in real time during a live broadcast,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, wherein the display format of the subtitles includes color, font size, and position.

[0866] (Claim 3)

[0867] The system according to claim 1, which generates and proposes insertion of relevant visual information and figures based on the analysis of the aforementioned video information.

[0868] "Application example 2 when combining with an emotional engine"

[0869] (Claim 1)

[0870] A means of analyzing acoustic information in real time and converting it into textual information,

[0871] A means for translating the aforementioned textual information into multiple languages,

[0872] A method for analyzing video information and proposing a subtitle display format,

[0873] A means for automatically generating subtitles using translated characters from the aforementioned multiple languages,

[0874] Means for allowing the user to edit the display format of the subtitles,

[0875] A means for analyzing the emotional state of a user obtained from video and audio information,

[0876] A means of dynamically adjusting the subtitle display style according to the user's emotional state,

[0877] A means of creating relevant visual information using a generative AI model and suggesting its display along with subtitles as needed,

[0878] A means of displaying the aforementioned subtitles in real time during a live broadcast,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, wherein the display style of the subtitles includes color, font size, and position, and is adjusted based on the user's emotional state.

[0882] (Claim 3)

[0883] The system according to claim 1, which generates and proposes insertion of relevant visual information based on the aforementioned video information and the user's emotional state. [Explanation of Symbols]

[0884] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of analyzing audio data in real time and converting it into text data, A means for translating the aforementioned text data into multiple languages, A method for analyzing video data and proposing subtitle display styles, A means for automatically generating subtitles using the translated texts of the aforementioned multiple languages, A means to allow the user to edit the display style of the aforementioned subtitles, A means of displaying the aforementioned subtitles in real time during a live broadcast, A system that includes this.

2. The system according to claim 1, wherein the display style of the subtitles includes color, font size, and position.

3. The system according to claim 1, which generates and proposes the insertion of related image diagrams and illustrations based on the analysis of the aforementioned video data.

Images