Electronic device and audio track obtaining method thereof

Abstract

An electronic device is disclosed. The electronic device may include: a display; a speaker; a memory storing at least one instruction; and at least one processor. The at least one processor may control the display to display video data including subtitle data of a target language, and obtain context information of a speaker from the video data and audio data corresponding to the video data; based on the subtitle data and the context information of the speaker, audio track data corresponding to the target language is obtained.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a method for obtaining an audio track, and more particularly, to an electronic device for obtaining an audio track based on subtitle data and a method for obtaining the audio track thereof. Background Art

[0002] Due to the development of electronic technology, various types of electronic devices have been developed and distributed. In particular, the distribution of display devices such as TVs or smartphones is actively progressing.

[0003] According to research, in 2023, moving images will account for 82% or more of all consumer Internet traffic, and there is a higher probability that 80% of users will watch the entire video if subtitles / closed captions are provided. Subtitles are the key to being seen by more users and can play a major role in image proximity. Having subtitles on moving images increases viewer engagement, improves user satisfaction, and gives the user choice in how he or she views moving images. According to a Verizon report, 50% or more people prefer subtitles, and 80% of users are more likely to watch the entire video when provided with subtitles. As the reliance on subtitles increases, the size of the global subtitle and captioning solutions market is expected to grow at a compound annual growth rate (CAGR) of 7.4% from $263.4 million in 2019 to $351 million in 2025.

[0004] Therefore, the demand for translation capabilities for video streaming is increasing. Localization of video content is the process used to adapt marketing content to foreign markets, and subtitles are often the easiest method for localizing videos.

[0005] Compared to subtitles, providing an audio track of subtitles as speech makes it easier for users to watch video content and can help visually impaired people watch movies / TV content with subtitles. However, the number of available audio track options is usually much smaller than the number of available subtitle languages. Summary of the invention

[0006] Technical Solution

[0007] According to one aspect of the present disclosure, an electronic device includes: a display; a speaker; a memory storing one or more instructions; and one or more processors operably coupled to the display, the speaker and the memory and configured to execute the one or more instructions, wherein the one or more instructions, when executed by the one or more processors, cause the electronic device to perform the following operations: control the display to display video data including subtitle data in a target language; obtain speaker context information from the video data and audio data corresponding to the video data; obtain audio track data corresponding to the target language based on the subtitle data and the speaker context information; and control the speaker to output the obtained audio track data.

[0008] The one or more instructions, when executed by the one or more processors, may cause the electronic device to perform the following operations: input the subtitle data and the context information of the speaker into a trained artificial intelligence (AI) model to obtain the audio track data, and the trained AI model may be a text-to-speech (TTS) AI model, wherein the TTS AI model is trained to receive text data as input and convert the text data into speaker-adapted audio data based on the context information of the speaker.

[0009] The trained AI model may be configured to obtain characteristic parameters of the speaker based on contextual information of the speaker, and output audio track data, in which the subtitle data is converted into the speaker-adaptive audio data based on the characteristic parameters of the speaker.

[0010] When executed by the one or more processors, the one or more instructions may cause the electronic device to perform the following operations: identify characteristic parameters of the speaker based on context information of the speaker, and convert the subtitle data into the speaker-adapted audio data based on the characteristic parameters of the speaker to obtain the audio track data.

[0011] The characteristic parameters of the speaker may include at least one of voice type, voice intonation, voice pitch, voice speed or voice volume, and the context information of the speaker may include at least one of the speaker's gender information, age information, emotion information, role information or speaking volume information.

[0012] The one or more instructions, when executed by the one or more processors, may cause the electronic device to perform the following operations: obtain timing data related to the start of the speaker's speech, recognition data of the speaker, and emotion data of the speaker based on at least one of the video data or the audio data, and identify characteristic parameters of the speaker based on the timing data, the recognition data, and the emotion data.

[0013] The one or more instructions, when executed by the one or more processors, may cause the electronic device to perform the following operations: obtain the subtitle data streamed from a specific data channel, or obtain the subtitle data through text recognition for a frame included in the video data.

[0014] The one or more instructions, when executed by the one or more processors, may cause the electronic device to perform the following operations: separate the audio data corresponding to the video data into background audio data and speech audio data, obtain context information of the speaker from the speech audio data, control the speaker to mix the audio track data and the background audio data obtained based on the subtitle data and the context information of the speaker, and output the mixed audio track data and background audio data.

[0015] The one or more instructions, when executed by the one or more processors, may cause the electronic device to perform the following operations: based on the video data including the first speaker and the second speaker, obtain first timing data related to the start of speaking of the first speaker and second timing data related to the start of speaking of the second speaker; based on the first timing data, the first recognition data of the first speaker and the first emotion data of the first speaker, perform text-to-speech (TTS) conversion on first subtitle data corresponding to the first speaker to obtain first audio track data corresponding to the first speaker; and based on the second timing data, the second recognition data of the second speaker and the second emotion data of the second speaker, perform TTS conversion on second subtitle data corresponding to the second speaker to obtain second audio track data corresponding to the second speaker.

[0016] According to one aspect of the present disclosure, a method for obtaining an audio track performed by an electronic device includes: displaying video data including subtitle data in a target language via a display of the electronic device; obtaining context information of a speaker from the video data and audio data corresponding to the video data; obtaining audio track data corresponding to the target language based on the subtitle data and the context information of the speaker; and outputting the obtained audio track data via a speaker of the electronic device.

[0017] According to one aspect of the present disclosure, a non-transitory computer-readable medium storing instructions, which, when executed by a processor of an electronic device, causes the electronic device to perform the following operations: display video data including subtitle data in a target language via a display of the electronic device; obtain context information of a speaker from the video data and audio data corresponding to the video data; obtain audio track data corresponding to the target language based on the subtitle data and the context information of the speaker; and output the obtained audio track data via a speaker of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a view for illustrating one or more embodiments of an electronic device according to one or more embodiments;

[0019] Figure 2a is a block diagram showing a configuration of an electronic device according to one or more embodiments;

[0020] Figure 2b is a view showing a detailed configuration of an example of an electronic device according to one or more embodiments;

[0021] Figure 3 is a view showing an audio track obtaining method according to one or more embodiments;

[0022] Figure 4 is a view showing an audio track obtaining method according to one or more embodiments;

[0023] Figure 5 It specifically shows Figure 4 A view of the audio track obtaining method shown;

[0024] Figure 6a and Figure 6b is a view illustrating a method of capturing a screen to obtain subtitle text data according to one or more embodiments;

[0025] Figure 7 is a view showing an audio track obtaining method according to one or more embodiments;

[0026] Figure 8 is a view showing an implementation form of a TTS module according to one or more embodiments;

[0027] Fig. 9 is a view showing a method of obtaining recognition data of a speaker according to one or more embodiments;

[0028] Fig.10 is a view showing a method of obtaining emotion data of a speaker according to one or more embodiments;

[0029] Figures 11a to 11d is a view showing an example of obtaining context information of a speaker according to one or more embodiments;

[0030] Figure 12 to Figure 14 is a diagram illustrating a use case according to one or more embodiments; and

[0031] Fig.15 is a view illustrating an infringement disclosure method according to one or more embodiments. DETAILED DESCRIPTION

[0032] Hereinafter, example embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0033] The terms used in the specification are briefly described, and then the present disclosure is described in detail.

[0034] Considering the functions in the present disclosure, the terms used in the embodiments of the present disclosure are selected as general terms that are used as widely as possible, but may vary according to the intentions of those skilled in the art or precedents, the emergence of new technologies, etc. In one or more examples, there are terms that the applicant arbitrarily selects in a specific case, and in this case, their meanings will be specifically described in the relevant description section of the present disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and throughout the entire content of the present disclosure, rather than based on the simple names of the terms.

[0035] In the specification, expressions such as “have”, “may have”, “include” or “may include” indicate the existence of such features (for example, numerical values, functions, operations, and components such as parts), and do not exclude the existence of additional features.

[0036] The expression “at least one of A and / or B” should be interpreted as meaning any one of “A” or “B” or “A and B”.

[0037] The expressions "1st", "2nd", "first", "second", etc. used in the specification may be used to describe various elements without regard to any order and / or degree of importance, wherein such expressions are merely used to distinguish one element from another and are not intended to limit the relevant elements.

[0038] The description that one element (e.g., a first component) is “(operably or communicatively) coupled to” / “(operably or communicatively) coupled to” or “connected to” another element (e.g., the second component) should be interpreted such that one element can be directly coupled to another element, or one element can be coupled to another element through another element (e.g., a third component).

[0039] Unless clearly defined differently in the context, a singular expression includes a plural expression. In the present application, terms such as "comprising" or "consisting of" should be interpreted as specifying the existence of such features, numbers, steps, operations, components, parts or combinations thereof described in the specification, but do not exclude the possibility of pre-existence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0040] In the present disclosure, a "module" or "component" may perform at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. In one or more examples, multiple "modules" or "components" may be integrated into at least one module and implemented as at least one processor (not shown), excluding "modules" or "components" that need to be implemented as specific hardware.

[0041] Hereinafter, one or more embodiments of the present disclosure are described in more detail with reference to the accompanying drawings.

[0042] Figure 1 is a view for illustrating one or more embodiments of an electronic device according to one or more embodiments.

[0043] The electronic device 100 may be implemented as an augmented reality (AR) / virtual reality (VR) device, such as a television (TV), a smart phone, a tablet personal computer (PC), or a head mounted display (HMD). Figure 1 As shown, but not limited to, the electronic device 100 can be applied to any device with a display function, such as a laptop PC, a video wall, a large format display (LFD), a digital signage, a digital information display (DID), a projector display, a near-eye display (NED), an electronic photo frame, a camera, or a camcorder, without limitation.

[0044] According to one or more embodiments, the electronic device 100 may receive and display video data having various compressed images and / or various resolutions in a streaming method. In one or more examples, the electronic device is not limited thereto, and may display downloaded video data. For example, the electronic device 100 may receive images in a compressed form, such as a Moving Picture Experts Group (MPEG) (e.g., MP2, MP4, MP7, etc.), a Joint Photographic Experts Group (JPEG), an Advanced Video Coding (AVC), H.264, H.265, or a High Efficiency Video Codec (HEVC). Otherwise, the electronic device 100 may receive any one of a standard definition (SD) image, a high definition (HD) image, a full HD image, or an ultra HD image.

[0045] For example, the electronic device 100 may display video data including streaming video data, in particular, video data including subtitle data together. Subtitle data represents text data in which voice audio data is translated into a language understandable to a user. According to one or more embodiments, subtitle data may be included in an area of ​​a frame included in the video data and received. According to another example, the electronic device 100 may synchronize subtitle data received through a separate data channel with audio data corresponding to the video data and display the subtitle data.

[0046] According to one or more embodiments, the electronic device 100 may output voice data in a language corresponding to the subtitle data by generating a real-time audio track for the subtitle data. Specifically, the electronic device 100 may generate and output audio track data of a voice on which the context of the original video data and / or the original audio data is reflected, rather than audio track data of a fixed voice. If video data is received in real-time streaming, audio track data may also be generated and output in real time. According to one or more embodiments, if the electronic device includes a speaker, the electronic device 100 may output the audio track data through the speaker, and if the electronic device does not include a speaker, the electronic device 100 may send the audio track data to an external speaker device.

[0047] Hereinafter, various embodiments of generating audio track data of a voice on which the context of original video data and / or original audio data is reflected based on subtitle data are described.

[0048] Figure 2a is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0049] according to Figure 2a , the electronic device 100 includes a display 110 , a speaker 120 , a memory 130 , and one or more processors 140 .

[0050] The display 110 may be implemented as a display including a spontaneous emission element or a display including a non-spontaneous emission element and a backlight. For example, the display may be implemented as various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a light emitting diode (LED), a micro-LED, a mini-LED, a plasma display panel (PDP), a quantum dot (QD) display, and a quantum dot light emitting diode (QLED). The display 110 may include a driving circuit, which may be implemented in a form such as an a-siTFT, a low temperature polycrystalline silicon (LTPS) TFT or an organic TFT (OTFT), a backlight unit, etc. In one or more examples, the display 110 may be implemented as a flexible display, a rollable display, a 3D display, a display in which multiple display modules are physically connected, and the like.

[0051] The speaker 120 outputs audio data. For example, the speaker 120 may convert digital audio data processed in one or more processors 140 into analog audio data, and then amplify and output the converted analog audio data. For example, the speaker 120 may include at least one speaker unit configured to output at least one channel, a D / A converter, an audio amplifier, etc. According to one or more embodiments, the speaker 120 may be implemented to output various multi-channel acoustic signals. For example, the one or more processors 140 may control the speaker 120 to enhance processing and output the generated audio track data. According to one or more embodiments, the electronic device 100 may not include a speaker, and may send the audio track data to an external speaker device to output the generated audio track data through the external speaker device.

[0052] The memory 130 is electrically connected to one or more processors 140 and stores data required by various embodiments of the present disclosure. The memory 130 may be implemented as a memory embedded in the electronic device 100 according to the purpose of data storage, or may be implemented as a memory that can be removed from the electronic device 100. For example, data for driving the electronic device 100 is stored in a memory embedded in the electronic device 100, and data for the extended functions of the electronic device 100 may be stored in a memory that can be removed from the electronic device 100. In one or more examples, the memory embedded in the electronic device 100 may be implemented as at least one of a volatile memory (e.g., a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), etc.) or a non-volatile memory (e.g., a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., a NAND flash memory, a NOR flash memory, etc.), a hard disk drive, or a solid state drive (SSD)). In one or more examples, the memory that is removable from the electronic device 100 can be implemented as a memory card (e.g., a compact flash (CF) card, a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, a multimedia card (MMC)), etc.), an external memory that can be connected to a USB port (e.g., a USB memory), etc.

[0053] According to one or more embodiments, the memory 130 may store a computer program including at least one instruction or a plurality of instructions for controlling the electronic device 100 .

[0054] According to one or more embodiments, the memory 130 may store images (i.e., input images, various data, information, etc.) received from an external device (e.g., a source device), an external storage medium (e.g., a USB), an external server (e.g., a network hard disk), etc.

[0055] According to one or more embodiments, the memory 130 may store information about an AI model (or a neural network model) including a plurality of layers. In one or more examples, storing information about the AI ​​model may mean storing various information related to the operation of the AI ​​model, for example, information about a plurality of layers included in the AI ​​model, and information about parameters used in each of the plurality of layers (e.g., filter coefficients, biases, etc.).

[0056] According to one or more embodiments, the memory 130 may store various information required for image quality processing, for example, information for performing at least one of noise reduction, detail enhancement, tone mapping, contrast enhancement, color enhancement or frame rate conversion, algorithms and image quality parameters. In one or more examples, the memory 130 may store a final output image generated by image processing.

[0057] According to one or more embodiments, the memory 130 may be implemented as a single memory storing data generated from various operations according to the present disclosure. In one or more examples, according to another embodiment, the memory 130 may be implemented as a plurality of memories including storing each different type of data or storing each data generated from different steps.

[0058] In an embodiment of the present disclosure, it is described that various data are stored in an external memory 130 of one or more processors 140, however, as understood by a person of ordinary skill in the art, according to at least one embodiment, at least a portion of the data may be stored in a memory within a processor 140 in the electronic device 100 or one or more processors 140.

[0059] The one or more processors 140 may perform operations of the electronic device 100 according to various embodiments by executing at least one instruction stored in the memory 130 .

[0060] One or more processors 140 generally control the operation of the electronic device 100. Specifically, the processor 140 may be connected to each component of the electronic device 100 to generally control the operation of the electronic device 100. For example, the processor 140 may be electrically connected to the display 110 and the memory 130 to generally control the operation of the electronic device 100. The processor 140 may be configured as one processor or a plurality of processors.

[0061] According to an embodiment of the present disclosure, functions related to AI may be operated through one or more processors 140 and memory 130 .

[0062] The one or more processors 140 may be composed of one or more processors. At this time, the one or more processors 140 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU), but are not limited to the aforementioned examples of the processor.

[0063] The CPU is a general-purpose processor that can perform not only general operations but also AI operations, and can efficiently execute complex programs through a multi-layer cache structure. The CPU is advantageous for a serial processing method that can organically connect the previous calculation result and the next calculation result through sequential calculation. The general-purpose processor is not limited to the aforementioned example, excluding the case where the present disclosure specifies it as the aforementioned CPU.

[0064] A GPU is a processor for large-scale operations (such as floating-point operations for graphics processing), and can integrate cores on a large scale to perform large-scale operations in parallel. Specifically, compared with a CPU, a GPU may be advantageous for parallel processing methods, such as convolution operations. In one or more examples, a GPU may be used as a coprocessor to supplement the functions of a CPU. The processor for large-scale operations is not limited to the above examples, excluding the case where the present disclosure specifies it as the above-mentioned GPU.

[0065] NPU is a processor dedicated to AI operations using artificial neural networks, where each layer configuring the artificial neural network can be implemented as hardware (e.g., silicon). In one or more examples, the NPU can be designed to be specific to the specifications required by the manufacturer, so the degree of freedom of the NPU is lower than that of the CPU or GPU, but the AI ​​operations required by the manufacturer can be effectively performed. In one or more examples, as a processor specific to AI operations, the NPU can be implemented in various forms, such as a tensor processing unit (TPU), an intelligent processing unit (IPU), or a visual processing unit (VPU). The AI ​​processor is not limited to the above examples, excluding the case where the present disclosure specifies it as the above-mentioned NPU.

[0066] In one or more examples, the one or more processors 140 may be implemented as a system on chip (SoC). In one or more examples, in addition to the one or more processors 140, the SoC may also include a memory and a network interface (such as a bus for data communication between the processor and the memory).

[0067] If the SoC included in the electronic device 100 includes a plurality of processors, the electronic device may perform AI-related operations (e.g., operations related to learning or inference of an AI model) by using some of the plurality of processors. For example, the electronic device 100 may perform AI-related operations by using at least one of a GPU, an NPU, a VPU, a TPU, or a hardware accelerator dedicated to AI operations (such as convolution operations or matrix product calculations) among the plurality of processors. The above description is merely an example, and as understood by those of ordinary skill in the art, AI-related operations may be processed by using a general-purpose processor (such as a CPU).

[0068] In one or more examples, the electronic device 100 can perform operations for AI-related functions by using multiple cores (e.g., dual cores, quad cores) included in one processor. Specifically, the electronic device can perform AI operations such as convolution operations and matrix product calculations in parallel by using multiple cores included in the processor.

[0069] For ease of description, one or more processors 140 may be referred to as processor 140 .

[0070] Figure 2b is a view illustrating a detailed configuration of an example of an electronic device according to one or more embodiments.

[0071] according to Figure 2b , the electronic device 100 ′ includes a display 110, a speaker 120, a memory 130, one or more processors 140, a communication interface 150, a user interface 160, and a camera 170. Figure 2b The components shown in Figure 2a Detailed description of components that are repeated in the illustrated components.

[0072] The communication interface 150 may perform communication with an external device. The communication interface 150 may receive an input image from an external device (e.g., a source device), an external storage medium (e.g., a USB memory), an external server (e.g., a network hard disk), etc. in a streaming or downloading method through a communication method (such as AP-based Wi-Fi (Wi-Fi, wireless LAN), Bluetooth, Zigbee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE1394, mobile high-definition link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU), an optical method, or a coaxial method). In one or more examples, the input image may be any one of a standard definition (SD) image, a high definition (HD) image, a full HD image, or an ultra HD image, but is not limited thereto.

[0073] The user interface 160 may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-mentioned display function and the manipulation input function together. According to one or more embodiments, the user interface 160 may be implemented as a transceiver of a remote controller, and may receive a remote control signal. The transceiver of the remote controller may receive a remote signal from an external remote control device or send a remote signal to an external remote control device through at least one communication method of an infrared communication method, a Bluetooth communication method, or a Wi-Fi communication method.

[0074] The camera 170 may be turned on and captured according to a preset event. The camera 170 may convert the captured image into an electrical signal and generate image data based on the converted signal. For example, an object may be converted into an electrical image signal by a semiconductor optical device, a charge coupled device (CCD), and the image signal converted as above may be amplified and converted into a digital signal, and then signal processing may be performed.

[0075] According to one or more embodiments, the electronic device 100' may further include a tuner and a demodulator. The tuner may tune a channel selected by a user in a radio frequency (RF) broadcast signal received through an antenna or all pre-stored channels to receive the RF broadcast signal. The demodulator may receive and demodulate a digital IF (DIF) signal converted in the tuner, and may perform channel demodulation, etc.

[0076] Figure 3 is a view illustrating an audio track obtaining method according to one or more embodiments. Figure 3 The method shown can be performed by the electronic device 100 ( Figure 2a ) or electronic device 100'( Figure 2b )implement.

[0077] According to one or more embodiments, Figure 3 As shown, the processor 140 may control the display 110 to display the video data including the subtitle data in the target language (S310). For example, the processor 140 may receive and display the video data including the subtitle data, or may synchronize the subtitle data received through a separate data channel with the audio data corresponding to the video data and display the subtitle data.

[0078] Then, the processor 140 may obtain context information of the speaker included in the video data from at least one of the video data or the audio data corresponding to the video data. In one or more examples, the context information of the speaker may include at least one of the gender information, age information, emotion information, role information, or speaking volume information of the speaker.

[0079] Then, the processor 140 may obtain audio track data corresponding to the target language based on the subtitle data and the context information of the speaker. In one or more examples, the audio track may be a track in which an audio signal is digitally recorded. For example, the processor 140 may obtain not only text data in the target language corresponding to the subtitle data obtained based on the audio data / video data, but also audio track data including or indicating metadata (gender, age, emotion, etc.) of the speaker.

[0080] Thereafter, the processor 140 may control the speaker 120 to output the obtained audio track data.

[0081] Figure 4 is a view illustrating an audio track obtaining method according to one or more embodiments.

[0082] Figure 4 The processor 140 shown in the figure can generate an audio track by using a subtitle data acquisition module 141, a speaker's context data acquisition module 142, and a TTS module 143. In one or more examples, each module can be implemented as at least one software, at least one hardware, and / or a combination thereof. For example, at least one of the subtitle data acquisition module 141, the speaker's context data acquisition module 142, and the TTS module 143 can be implemented as using a trained AI model and / or a predefined algorithm. According to one or more embodiments, the subtitle data acquisition module 141, the speaker's context data acquisition module 142, and the TTS module 143 can be included in the electronic device 100, but can be distributed in at least one server. For example, the subtitle data acquisition module 141 can be included in the first server, the speaker's context data acquisition module 142 can be included in the second server, and the TTS module 143 can be included in the third server. In one or more examples, the first server to the third server can be directly connected to each other, or can be connected through the electronic device 100 (or a relay server) to send and receive data.

[0083] According to one or more embodiments, the processor 140 may obtain subtitle data streamed from a specific data channel by using the subtitle data obtaining module 141, or obtain subtitle data by text recognition for a frame included in video data. For example, the processor 140 may obtain subtitle data from a digital stream streamed from a specific data channel.

[0084] According to one or more embodiments, the processor 140 may obtain context information of the speaker from the video data and the audio data by using the context data obtaining module 142 of the speaker. In one or more examples, the context information of the speaker may include at least one of the gender information, age information, emotion information, role information, or speaking volume information of the speaker. In one or more examples, the context information may be included in metadata associated with the video data. In one or more examples, the processor 140 may perform image recognition on one or more video frames included in the video data to obtain the context information.

[0085] According to one or more embodiments, the processor 140 may convert text data corresponding to the subtitles (e.g., text data in a target language) into speech by using the TTS module 143. In this case, the TTS module 143 may not convert the text data corresponding to the subtitles into a predefined speech, but may convert it into a speech reflecting the context of the speaker. For example, the TTS module 143 may obtain audio track data reflecting the gender information, age information, emotion information, role information, or speaking volume information of the speaker. The TTS module 143 may also obtain audio track data of a pitch / tone as similar as possible to the pitch of the speaker based on information about the speech obtained based on the audio data (e.g., frequency information (frequency band, spectrum, frequency waveform, etc.)).

[0086] Figure 5 It specifically shows Figure 4 A view of the audio track obtaining method shown in .

[0087] according to Figure 5 In one or more embodiments shown, the context data acquisition module 142 (hereinafter, the context data acquisition module) of the speaker can identify the data of the speaker, for example, the context data based on the audio data and the video data. In one or more examples, the audio data can be data synchronized with the video data and received, and can be data that is partially pre-processed and / or post-processed, but is not limited thereto. For example, if the encoded audio data and / or video data is received, it can be used to obtain the context information of the speaker after decoding. For example, the audio data and / or video data can be data stored in the output buffer before being decoded and output. However, the embodiments are not limited to these configurations. For example, after the audio data and / or video data buffered in the receiving buffer are decoded, the audio data and / or video data can be stored in a separate memory instead of being stored in the output buffer for the context information of the speaker. In one or more examples, the present disclosure is not limited to these configurations, and the screen can be captured in a state where the video data is displayed on the screen and the subtitle data included in the captured screen can be identified to obtain the subtitle text data.

[0088] According to one or more embodiments, the context data acquisition module 142 may identify context data of the speaker, including timing data related to the start of the speaker's speech from the video data and the audio data, identification data of the speaker, and emotion data of the speaker. In one or more examples, the identification data of the speaker may include various types of identification data that may identify the speaker, such as gender information, age information, role information, or actor information of the speaker. The emotion data of the speaker may include various types of emotion data, such as joy, sadness, anger, surprise, boredom, and calmness.

[0089] According to one or more embodiments, the context data acquisition module 142 may obtain each context data related to each speaker (e.g., a first speaker and a second speaker). The context data acquisition module 142 may obtain first timing data related to the start of speaking of the first speaker, first recognition data of the first speaker, and first emotion data of the first speaker, and may obtain second timing data related to the start of speaking of the second speaker, second recognition data of the second speaker, and second emotion data of the second speaker. According to one or more embodiments, the context data acquisition module 142 may analyze features (e.g., intonation, pitch, or frequency) of speech data included in the audio data, identify whether the speech data is the speech of the same speaker or the speech of another speaker, and obtain the first timing data and the second timing data based on the identification.

[0090] In one or more examples, the context data acquisition module 142 may obtain the characteristic parameters of the speaker based on the context information of the speaker. In one or more examples, the characteristic parameters of the speaker may include at least one of the voice type, voice intonation, voice pitch, voice speed, or voice volume. In one or more examples, the characteristic parameters of the voice may also include information such as the frequency band, spectrum, or frequency waveform of the voice.

[0091] The subtitle data acquisition module 141 can obtain subtitle text data by text recognition for frames included in the video data. For example, the processor 140 can obtain subtitle data by text area (or subtitle area) recognition and optical character reader (OCR) recognition for video frames. For example, the processor 140 can input video data into a trained AI model (or algorithm) to obtain subtitle data. In one or more examples, the trained AI model can be trained to detect, recognize and output text data included in the video data.

[0092] According to one or more embodiments, if subtitle data is streamed from a specific data channel as a digital stream, the subtitle data may be provided to the TTS module 143 without passing through the subtitle data obtaining module 141. In one or more examples, since synchronization between the speaker's feature data and the subtitle data is required even in this case, the subtitle data obtaining module 141 may perform a related function.

[0093] The TTS module 143 may generate speaker-adaptive audio track data based on the subtitle text data and the role parameters of the speaker. According to one or more embodiments, the subtitle text data output from the subtitle data acquisition module 141 and / or the subtitle text data streamed from a specific data channel may be synchronized with the characteristic parameters of the speaker output from the context data acquisition module 142 to be input into the TTS module 143. In this case, synchronization may be performed at the output end of the subtitle data acquisition module 141 and the context data acquisition module 142, or synchronization may be performed at the input end of the TTS module 143. For example, the subtitle text data and the characteristic parameters of the speaker may be synchronized based on the timing data related to the start of speaking obtained from the video data and the audio data, and may be input into the TTS module 143.

[0094] According to one or more embodiments, if the video data includes multiple speakers (e.g., a first speaker and a second speaker), the TTS module 143 may convert the corresponding subtitle text based on the context data of each speaker. For example, the TTS module 143 performs TTS conversion on the first subtitle text data corresponding to the first speaker based on the first timing data, the first recognition data of the first speaker, and the first emotion data of the first speaker to obtain the first audio track data corresponding to the first speaker. In one or more examples, the TTS module 143 may perform TTS conversion on the second subtitle data corresponding to the second speaker based on the second timing data, the second recognition data of the second speaker, and the second emotion data of the second speaker to obtain the second audio track data corresponding to the second speaker.

[0095] Figure 6a and Figure 6b is a view illustrating a method of capturing a screen to obtain subtitle text data according to one or more embodiments.

[0096] According to one or more embodiments, the subtitle data obtaining module 141 may identify subtitle data included in video data to obtain subtitle text data.

[0097] according to Figure 6aIn one or more embodiments shown in , the subtitle data acquisition module 141 may capture a screen (141-1), and if it is identified that a subtitle area is included in the captured screen image (S141-2), the position of the subtitle area (141-3) may be identified. Then, the subtitle data acquisition module 141 may perform text recognition (e.g., OCR recognition) at the identified position of the subtitle area (141-4) to obtain text. Thereafter, the subtitle data acquisition module 141 may obtain a final subtitle text (141-6) by performing grammar correction on the obtained text (141-5). For example, as Figure 6b As shown, the subtitle data obtaining module 141 may capture the screen while the video data is displayed on the display 110 , and may obtain subtitle text through text recognition with respect to the subtitle regions 610 , 620 included in the captured screen image.

[0098] Figure 7 is a view illustrating an audio track obtaining method according to one or more embodiments.

[0099] according to Figure 7 , the processor 140 may generate an audio track by using the subtitle data obtaining module 141, the speaker's context data obtaining module 142, the TTS module 143, the adaptive filter module 144, and the audio mixer 145. Figure 7 The components shown are Figure 4 and Figure 5 Detailed descriptions of components are repeated for components shown. In one or more examples, each module can be implemented as at least one software, at least one hardware, and / or a combination thereof. Figure 5 The subtitle data obtaining module 141 is fully described in , and therefore, its illustration will be omitted for ease of description.

[0100] In one or more examples, the processor 140 may separate the audio data into background audio data and speech audio data by using the adaptive filter module 144. The background audio data separated by the adaptive filter module 144 may be provided to the audio mixer 145, and the speech audio data may be provided to the context data acquisition module 142 of the speaker.

[0101] According to one or more embodiments, the context data acquisition module 142 may identify context data of the speaker, including timing data related to the start of speech from the video data and the speech audio data, identification data of the speaker, and emotion data of the speaker. In one or more examples, the context data acquisition module 142 may obtain characteristic parameters of the speaker based on the context information of the speaker. In one or more examples, the characteristic parameters of the speaker may include at least one of a voice type, a voice intonation, a voice pitch, a voice speed, or a voice volume. In addition, the characteristic parameters of the speaker may also include information such as a frequency band, a spectrum, or a frequency waveform of the voice.

[0102] The TTS module 143 may generate speaker-adaptive audio track data based on the subtitle text data and the speaker's role parameters. According to one or more embodiments, the TTS module 143 may adjust the speaker's feature parameters (143-1) based on the timing data, the speaker's recognition data, and the speaker's emotion data, and perform TTS on the subtitle text data based on the adjusted parameters to generate speech data corresponding to the subtitle text.

[0103] According to one or more embodiments, the TTS module 143 may adjust at least one of a voice type, a voice intonation, a voice pitch, a voice rate, or a voice volume based on the timing data, the speaker's recognition data, and the speaker's emotion data.

[0104] Thereafter, the audio mixer 145 may output new audio data, that is, audio track data, by mixing the background audio data and the speech data output from the TTS module 143 .

[0105] Figure 8 is a view showing an implementation form of a TTS module according to one or more embodiments.

[0106] According to one or more embodiments, the TTS module may be implemented as a stored and predefined operating rule or AI model for processing input data. The predefined operating rule or AI model is constructed by learning. In one or more examples, building by learning means constructing a predefined operating rule or AI model with desired characteristics by applying a learning algorithm to various learning data. The learning may be performed in the device itself that executes the AI ​​according to the present disclosure, and may also be performed by a separate server / system.

[0107] The AI ​​model may be composed of multiple neural network layers. At least one layer has at least one weight value, and the operation of the layer is performed by the operation result of the previous layer and at least one defined operation. Examples of neural networks are convolutional neural networks (CNNs), deep neural networks (DNNs), recursive neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recursive deep neural networks (BRDNNs), deep Q networks, and transformers, wherein the neural networks of the present disclosure are not limited to the above examples, excluding the case where the neural network is specified in the present disclosure. A learning algorithm may be a method in which a given target device (e.g., a robot) is trained by using multiple learning data so that the given target device itself can make or predict decisions. Examples of learning algorithms are supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, wherein, except for the case where the learning algorithm is specified in the present disclosure, the learning algorithm is not limited to the above examples.

[0108] According to one or more embodiments, the TTS module may be implemented as a trained AI model. In this case, the processor 140 may obtain audio track data by inputting the subtitle text data and the context information of the speaker into the trained AI model. In one or more examples, the trained AI model may be a TTS AI model that is trained to convert text data into speaker-adaptive audio data based on the context information of the speaker and output. According to one or more embodiments, the trained AI model may obtain the characteristic parameters of the speaker based on the context information of the speaker, and be trained to output audio track data, wherein the subtitle text data is converted into speaker-adaptive audio data based on the characteristic parameters of the speaker.

[0109] Fig. 9 is a view illustrating a method of obtaining recognition data of a speaker according to one or more embodiments.

[0110] According to one or more embodiments, the processor 140 can obtain a person (identity) ID from a video clip by using a feature extractor, a frame feature aggregation module, a multimodal fusion module, and a classifier. For example, after the face embedding can be collected by the attention VLAD, the processor 140 can converge with other modal embeddings through the multimodal fusion module.

[0111] The feature extractor may perform image preprocessing and / or feature extraction. Multimodal information (such as face, audio sound, body, text, face, and audio sound) may be obtained from the input video. Thereafter, the feature extractor may extract each embedding by using a pre-trained model.

[0112] The frame aggregation module can aggregate features of frames configuring a video. For example, a video may be configured by frames that change continuously. For example, if a face is detected in a specific frame, there is a possibility that faces with the same label are included in adjacent frames even though the angle, facial expression, or definition may be different. After measuring the face quality, in order to remove the complexity caused by the different number of faces in different clips, frame features can be aggregated based on NetVLAD. If an arbitrary number of features are input, the frame aggregation module can generate a fixed-length embedding as a video-level facial feature.

[0113] The multimodal fusion module can learn the weight values ​​of the multimodal features and rescale the multimodal features according to the weight values. This is because the multimodal features of the video clips can be complementary and overlapping.

[0114] The classifier can predict the person ID. Under the premise of focusing on person identification in supervised videos, the person ID can be predicted by using a multi-layer perceptron (MLP) as a classifier.

[0115] Fig.10 is a view illustrating a method of obtaining emotion data of a speaker according to one or more embodiments.

[0116] According to one or more embodiments, facial emotion recognition (FER) may be performed on a sequence of faces, facial component detection, feature extraction, and facial expression classification. Fig.10 As shown, in an input image (a), FER can detect facial regions and facial landmarks (b), extract spatial and temporal features from facial components (e.g., eyes and nose) and landmarks (c), and determine facial expressions based on one of the facial categories identified by using pre-trained pattern classifiers such as support vector machines (SVM), AdaBoost, and random forests (d). In one or more examples, the above are merely examples, and the speaker's emotion data may be obtained in various methods other than the above.

[0117] Figures 11a to 11d is a view illustrating an example of obtaining context information of a speaker according to one or more embodiments.

[0118] according to Fig.11a , the first context information 1110 may be obtained based on the first video data 1111 and / or the first audio data 1112. For example, the gender, age, emotion, whether or not the speaker is speaking, the speaking volume, etc. of each speaker among the multiple speakers (speaker 1, speaker 2) included in the first video data 1111 may be obtained. In one or more examples, since the first audio data 1112 is not detected in the relevant example, the first context information 1110 excluding whether or not the speaker is speaking may be obtained based on the first video data 1111.

[0119] according to Fig.11b , the second context information 1120 may be obtained based on the second video data 1121 and / or the second audio data 1122. For example, the gender, age, emotion, whether to speak, speaking volume, etc. of each speaker among the multiple speakers (speaker 1, speaker 2) included in the second video data 1121 may be obtained. In a related example, since it is determined that only speaker 2 is speaking, the context information of speaker 1 may be identified based only on the second video data 1121, and the context information of speaker 2 may be identified based on the second video data 1121 and the second audio data 1122.

[0120] according to Fig.11c , the third context information 1130 may be obtained based on the third video data 1131 and / or the third audio data 1132. In a related example, since it is determined that only speaker 1 is speaking, the context information of speaker 2 may be identified based only on the third context information 1130, and the context information of speaker 1 may be identified based on the third video data 1131 and the third audio data 1132.

[0121] according to Fig.11d , the fourth context information 1140 may be obtained based on the fourth video data 1141 and / or the fourth audio data 1142. In a related example, since it is determined that only speaker 2 is speaking, the context information of speaker 1 may be identified based only on the fourth video data 1141, and the context information of speaker 2 may be identified based on the fourth video data 1141 and the fourth audio data 1142.

[0122] In the above example, when speaker 1 and speaker 2 are speaking, as understood by a person of ordinary skill in the art, when speaker 1 and speaker 2 are speaking at the same time, the context information of each speaker can be identified by identifying each of the speech of speaker 1 and the speech of speaker 2 from the audio data based on feature parameters of the speech.

[0123] Figure 12 to Figure 14 is a diagram illustrating a use case according to one or more embodiments.

[0124] According to Fig.12 In one or more embodiments shown, an audio track that reflects the voice characteristics and emotions of the speaker can be generated through speaker recognition. Therefore, natural audio track generation is possible. In one or more examples, a user can define the characteristics of the speaker and / or the voice of the actor so that a specific role voice is pronounced in a defined voice on the synthesized audio track.

[0125] According to Fig.13aIn one or more embodiments shown, a visually impaired person may not be able to see specific detailed text displayed on the screen in real time, but may also be able to recognize the specific detailed text by using the audio track. Fig.13a One or more of the illustrated embodiments provide a mode in which a region from which speech is output is outlined or highlighted by using TTS in a state in which the original audio sound is suppressed or paused.

[0126] According to Fig.13b According to one or more embodiments shown, a visually impaired person may not be able to access data that is displayed in real time on a TV screen but not audibly. However, according to one or more embodiments, a mode for recognizing and outputting all text excluding subtitles in speech by using TTS may be provided. For example, if the user lowers or pauses the original audio sound and then returns to normal mode, all text excluding subtitles may be recognized and output in speech by using TTS.

[0127] According to Fig.14 One or more embodiments shown, if the content includes incorrect words or phrases, then it may be necessary to remove the incorrect words or phrases (e.g., parental control mode). In this case, the offending words or phrases may be shielded or filtered by using audio track synthesis.

[0128] Fig.15 is a view illustrating an infringement disclosure method according to one or more embodiments.

[0129] according to Fig.15 , if video content including subtitles is streamed to the device (S1510), it may be identified whether the subtitle type is a visual subtitle or a streaming subtitle (S1520). In one or more examples, the subtitle language may be a language different from a language provided through an existing audio track. Visual subtitles may be subtitles fixed to a frame of the video content, and streaming subtitles may be subtitles streamed separately from video data.

[0130] If the subtitle type is a visual subtitle, it may be identified whether detection and recognition of the subtitle is possible (S1530).

[0131] If detection and recognition of subtitles are possible (S1530: true), it may be identified whether the subtitles can be voiced by TTS (S1540).

[0132] In one or more examples, if the subtitle type is a streaming subtitle, it may be identified whether the subtitle can be voiced by TTS ( S1540 ).

[0133] If the subtitles can be voiced by TTS (S1540: true), it may be identified whether a feature parameter of the voice depends on the video content (S1550).

[0134] If the characteristic parameters of the speech depend on the video content (S1550: true), it can be considered that infringement is established (S1560). According to one or more embodiments of the present disclosure, this is because the subtitles can be voiced by reflecting the context of the video content in consideration of this situation.

[0135] In one or more examples, if the situation corresponds to at least one of a situation where detection and recognition of visual subtitles is impossible (S1530: False), a situation where subtitles cannot be voiced by TTS (S1540: False), or a situation where characteristic parameters of speech do not depend on video content (S1550: False), then it can be considered that the situation makes infringement not established. According to one or more embodiments of the present disclosure, this is because the situation can be ignored so that the subtitles can be voiced by reflecting the context of the video content.

[0136] According to various embodiments, an audio track on which the voice characteristics and emotions of the speaker are reflected can be generated through speaker recognition. Therefore, since natural audio track generation is possible, the user's viewing experience can be improved. In one or more examples, the user can enable the user to define the characteristics of the speaker and / or the voice of the actor so that a specific character voice is uttered in the defined voice on the synthesized audio track, and in this regard, the user's viewing experience can be improved.

[0137] In one or more examples, the aforementioned methods according to various embodiments may be implemented in the form of an application that can be installed in an existing electronic device. In one or more examples, at least a portion of the methods according to various embodiments of the present disclosure as described above may be performed by using an AI model based on deep learning (i.e., a learning network model).

[0138] In one or more examples, the aforementioned methods according to various examples of the present disclosure may be implemented using only a software upgrade or a hardware upgrade for an existing electronic device.

[0139] In one or more examples, various examples of the present disclosure as described above may be implemented by an embedded server included in the electronic device or an external server of the electronic device.

[0140] In one or more examples, according to one or more embodiments of the present disclosure, the various examples described above may be implemented as software including instructions stored in a machine (e.g., computer) readable storage medium. A machine refers to a device that calls instructions stored in a storage medium and can operate according to the called instructions, wherein the machine may include an electronic device (e.g., electronic device A) according to a disclosed embodiment. If the instruction is executed by a processor, the processor may directly execute the function corresponding to the instruction, or execute the function corresponding to the instruction by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. In one or more examples, the term "non-transitory" only means that the storage medium does not include a signal and is tangible, wherein the term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored in a storage medium.

[0141] In one or more examples, according to one or more embodiments of the present disclosure, the methods according to the various examples described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., playstore). TM ) Online distribution. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored in a storage medium, or may be temporarily generated in a storage medium (such as a manufacturer's server, an application store's server, or a memory of a forwarding server).

[0142] In one or more examples, each component (e.g., module or program) according to the various embodiments described above may be configured as a single item or multiple items, wherein some of the related subcomponents described above may be omitted, or another subcomponent may be further included in various embodiments. Primarily or additionally, some components (e.g., modules or programs) may be integrated into one project, and the functions implemented by each related component before integration may be performed identically or similarly. According to various embodiments, the operations performed by a module, program, or another component may be performed sequentially, in parallel, repeatedly, or heuristically, at least a portion of the operations may be performed in a different order or omitted, or another operation may be added.

[0143] Although example embodiments of the present disclosure are shown and described, it should be understood that the embodiments of the present disclosure are not limited to the aforementioned embodiments, and various modifications may be implemented by those skilled in the art without departing from the subject matter of the present disclosure as claimed in the scope of the claims, wherein these modifications should not be understood independently of the technical spirit or prospects of the present disclosure.

Claims

1. An electronic device, comprising: monitor; speaker; a memory storing one or more instructions; as well as one or more processors operably coupled to the display, the speaker, and the memory and configured to execute the one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the electronic device performs the following operations: controlling the display to display video data including subtitle data in a target language; Obtaining context information of a speaker from the video data and audio data corresponding to the video data; Based on the subtitle data and the context information of the speaker, obtaining audio track data corresponding to the target language; and The speaker is controlled to output the obtained audio track data.

2. The electronic device according to claim 1, wherein: The one or more instructions, when executed by the one or more processors, cause the electronic device to input the subtitle data and the contextual information of the speaker into a trained artificial intelligence (AI) model to obtain the audio track data, and Wherein, the trained AI model is a text-to-speech (TTS) AI model, wherein the TTS AI model is trained to receive text data as input and convert the text data into speaker-adaptive audio data based on contextual information of the speaker.

3. The electronic device as claimed in claim 2, wherein: The trained AI model is configured to: Based on the context information of the speaker, obtaining characteristic parameters of the speaker; and The audio track data is output, in which the subtitle data is converted into the speaker-adaptive audio data based on the characteristic parameters of the speaker.

4. The electronic device as claimed in claim 3, wherein: The one or more instructions, when executed by the one or more processors, cause the electronic device to perform the following operations: identify characteristic parameters of the speaker based on context information of the speaker, and convert the subtitle data into the speaker-adapted audio data based on the characteristic parameters of the speaker to obtain the audio track data.

5. The electronic device as claimed in claim 3, wherein: The characteristic parameter of the speaker includes at least one of voice type, voice intonation, voice pitch, voice speed or voice volume, The context information of the speaker includes at least one of the gender information, age information, emotion information, role information or speaking volume information of the speaker.

6. The electronic device as claimed in claim 4, wherein: The one or more instructions, when executed by the one or more processors, cause the electronic device to perform the following operations: Based on at least one of the video data or the audio data, obtaining timing data related to the start of speech of the speaker, identification data of the speaker, and emotion data of the speaker, and Characteristic parameters of the speaker are identified based on the timing data, the recognition data and the emotion data.

7. The electronic device as claimed in claim 1, wherein: The one or more instructions, when executed by the one or more processors, cause the electronic device to perform the following operations: obtain the subtitle data streamed from a specific data channel, or The subtitle data is obtained through text recognition with respect to a frame included in the video data.

8. The electronic device as claimed in claim 1, wherein: The one or more instructions, when executed by the one or more processors, cause the electronic device to perform the following operations: separating the audio data corresponding to the video data into background audio data and speech audio data, Obtaining context information of the speaker from the speech audio data; The speaker is controlled to mix the audio track data and the background audio data obtained based on the subtitle data and the context information of the speaker, and output the mixed audio track data and background audio data.

9. The electronic device as claimed in claim 1, wherein: The one or more instructions, when executed by the one or more processors, cause the electronic device to perform the following operations: Based on the video data including the first speaker and the second speaker, obtaining first timing data related to the start of speaking of the first speaker and second timing data related to the start of speaking of the second speaker, Based on the first timing data, the first recognition data of the first speaker, and the first emotion data of the first speaker, performing a text-to-speech (TTS) conversion on the first subtitle data corresponding to the first speaker to obtain first audio track data corresponding to the first speaker; as well as Based on the second timing data, the second recognition data of the second speaker, and the second emotion data of the second speaker, TTS conversion is performed on the second subtitle data corresponding to the second speaker to obtain second audio track data corresponding to the second speaker.

10. A method for obtaining an audio track performed by an electronic device, the method comprising: displaying, via a display of the electronic device, video data including subtitle data in a target language; Obtaining context information of a speaker from the video data and audio data corresponding to the video data; Based on the subtitle data and the context information of the speaker, obtaining audio track data corresponding to the target language; as well as The obtained audio track data is outputted via a speaker of the electronic device.

11. The method of claim 10, wherein: The obtaining of the audio track data comprises: inputting the subtitle data and the context information of the speaker into a trained AI model to obtain the audio track data, and Wherein, the trained AI model is a text-to-speech (TTS) AI model, wherein the TTS AI model is trained to receive text data as input and convert the text data into speaker-adaptive audio data based on contextual information of the speaker.

12. The method of claim 11, wherein: The trained AI model is configured to: Based on the context information of the speaker, obtaining characteristic parameters of the speaker; The audio track data is output, in which the subtitle data is converted into the speaker-adaptive audio data based on the characteristic parameters of the speaker.

13. The method of claim 12, wherein: The obtaining of the audio track data includes: identifying characteristic parameters of the speaker based on context information of the speaker, and converting the subtitle data into the speaker-adaptive audio data based on the characteristic parameters of the speaker to obtain the audio track data.

14. The method of claim 12, wherein: The characteristic parameter of the speaker includes at least one of voice type, voice intonation, voice pitch, voice speed or voice volume, The context information of the speaker includes at least one of the gender information, age information, emotion information, role information or speaking volume information of the speaker.

15. A non-transitory computer-readable medium storing instructions, which, when executed by a processor of an electronic device, cause the electronic device to: displaying video data including subtitle data in a target language; Obtaining context information of a speaker from the video data and audio data corresponding to the video data; Based on the subtitle data and the context information of the speaker, obtaining audio track data corresponding to the target language; as well as The obtained audio track data is output.

Images