Electronic device and image output method thereof

The electronic device analyzes audio and pixel information to identify and manage special effects, enhancing user experience through immersive and three-dimensional content output, addressing the lack of effective special effect handling in existing technologies.

WO2024242288A1PCT designated stage expired Publication Date: 2024-11-28SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/002288
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-02-22
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing electronic devices lack effective methods to enhance user experience by identifying and outputting special effects in content, such as visual and auditory effects, which are crucial for immersive experiences in gaming and movie content, and fail to efficiently manage these effects across different display layers and areas.

Method used

An electronic device with a processor that analyzes audio signals and pixel information to identify special effects, outputs them as separate layers, and uses a learned neural network model to determine the presence and maintenance of special effects, allowing for dynamic adjustment and transmission of special effect information to external devices.

Benefits of technology

This solution enhances user experience by providing immersive and three-dimensional effects by accurately identifying and managing special effects, improving content engagement without requiring special hardware, and allowing for software-based graphics calculations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024002288_28112024_PF_FP_ABST
    Figure KR2024002288_28112024_PF_FP_ABST
Patent Text Reader

Abstract

An electronic device is disclosed. This electronic device comprises: a display; a memory for storing at least one instruction; and one or more processors connected to the display and the memory so as to control the electronic device. The one or more processors execute the at least one instruction to: identify whether to provide a special effect on the basis of pixel information included in at least one image frame included in content, when the magnitude of an audio signal corresponding to the at least one image frame is identified to be equal to or greater than a threshold value; identify image information corresponding to the special effect in the at least one image frame when it is identified that the special effect is provided in the at least one image frame; and output a special effect image obtained on the basis of the identified image information through the display together with the at least one image frame.
Need to check novelty before this filing date? Find Prior Art

Description

Electronic device and image output method thereof

[0001] The present disclosure relates to an electronic device and an image output method thereof, and more particularly, to an electronic device that outputs content providing special effects and an image output method thereof.

[0002] Driven by continuous technological advancements, various types of electronic devices are being developed. In particular, display devices, used in a variety of settings, including homes, offices, and public spaces, have been continuously evolving for years.

[0003] Various methods are being researched to enhance the user experience of content. For example, various special effects are being introduced to enhance immersion by providing a sense of depth in various content, such as games and movies.

[0004] According to one embodiment, an electronic device includes: a display; at least one memory storing at least one command; and at least one processor connected to the display and the memory and controlling the electronic device. The at least one processor, by executing the at least one command, identifies whether a special effect is to be provided based on pixel information included in the at least one image frame when the size of an audio signal corresponding to at least one image frame included in content is identified as being greater than a threshold value, and identifies image information corresponding to the special effect in the at least one image frame when it is identified that a special effect is to be provided, and outputs a special effect image obtained based on the identified image information through the display together with the at least one image frame.

[0005] The one or more processors can control the display so that, by executing the at least one command, when the at least one image frame is displayed in an area smaller than the entire area of ​​the display, the special effect image is output to an area remaining in the display except for the area in which the at least one image frame is displayed.

[0006] The one or more processors can control the display by executing the at least one command to provide a first display layer including the at least one image frame and to provide a second display layer including the special effect image at a rear end of the first display layer.

[0007] The at least one processor can control the display by executing the at least one instruction to provide a first display layer including the at least one image frame, provide a third display layer including a frame graphic in front of the first display layer, remove a background image outside the frame graphic from the at least one image frame, and provide a second display layer including the special effect image in front of the third display layer.

[0008] The electronic device further includes a user interface, and the at least one processor, by executing the at least one command, when a user input is received through the user interface, identifies whether the user input is related to providing the special effect, and when the user input is identified as being related to providing the special effect, identifies whether the special effect is provided in the at least one image frame based on the user input.

[0009] The one or more processors can, by executing the at least one command, obtain color histogram information based on the pixel information, compare the color histogram information with reference color histogram information to obtain histogram difference information, and identify whether to provide the special effect or at least one of image information based on the histogram difference information.

[0010] The electronic device further includes a speaker; and the one or more processors, by executing the at least one command, when it is identified that a special effect is provided in the at least one image frame, can identify audio information corresponding to the special effect in the at least one image frame and output special effect audio through the speaker based on the audio information.

[0011] The one or more processors may, by executing the at least one command, input the at least one frame and the audio signal corresponding to the at least one image frame into a learned neural network model to obtain at least one of whether the special effect is provided, the image information, or the audio information corresponding to the special effect. The learned neural network model may be trained to obtain at least one of whether the special effect is provided, the image information, or the audio information by learning changes in the size and color of the audio signal.

[0012] The one or more processors can identify whether the special effect is maintained or terminated in the at least one subsequent frame based on a difference between the pixel information and pixel information included in the at least one subsequent frame by executing the at least one command, and identify a frame section that provides the special effect image based on whether the identified special effect is maintained or terminated.

[0013] The electronic device further includes a communication interface, wherein the one or more processors can transmit at least one of video information or audio information corresponding to the special effect to an external device through the communication interface, together with information about the content, by executing the at least one command. The information about the content may include identification information of the content and information about a playback time at which the special effect is provided.

[0014] A method for outputting an image of an electronic device according to an embodiment includes: a step of identifying whether a special effect is to be provided based on pixel information included in at least one image frame when a size of an audio signal corresponding to at least one image frame included in content is identified as being greater than a threshold value; a step of identifying image information corresponding to the special effect in the at least one image frame when a special effect is identified as to be provided in the at least one image frame; and a step of outputting a special effect image acquired based on the image information together with the at least one image frame.

[0015] The step of outputting together with the at least one image frame may include a step of outputting the special effect image in an area remaining in the display except for the area in which the at least one image frame is displayed, when the at least one image frame is displayed in an area smaller than the entire area of ​​the display.

[0016] The step of outputting together with the at least one image frame may include: providing a first display layer including the at least one image frame; and providing a second display layer including the special effect image displayed behind the first display layer.

[0017] The step of outputting together with the at least one image frame may include: providing a first display layer including the at least one image frame; providing a third display layer including a frame graphic on the front of the first display layer; removing a background image outside the frame graphic from the at least one image frame; and providing a second display layer including the special effect image on the front of the third display layer.

[0018] A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device according to one or more embodiments, cause the electronic device to perform an operation, the operation includes: when a size of an audio signal corresponding to at least one image frame included in content is identified as being greater than a threshold value, identifying whether a special effect is to be provided based on pixel information included in the at least one image frame; when a special effect is identified as to be provided in the at least one image frame, identifying image information corresponding to the special effect in the at least one image frame; and outputting a special effect image acquired based on the image information together with the at least one image frame.

[0019] The step of outputting together with the at least one image frame may include a step of outputting the special effect image in an area remaining in the display except for the area in which the at least one image frame is displayed, when the at least one image frame is displayed in an area smaller than the entire area of ​​the display.

[0020] The step of outputting together with the at least one image frame may include: providing a first display layer including the at least one image frame; and providing a second display layer including the special effect image displayed behind the first display layer.

[0021] The step of outputting together with the at least one image frame may include: providing a first display layer including the at least one image frame; providing a third display layer including a frame graphic on the front of the first display layer; removing a background image outside the frame graphic from the at least one image frame; and providing a second display layer including the special effect image on the front of the third display layer.

[0022] The above operation may include: a step of obtaining color histogram information based on the pixel information; a step of comparing the color histogram information with reference color histogram information to obtain histogram difference information; and a step of identifying whether to provide the special effect or at least one of image information based on the histogram difference information.

[0023] The above operation includes a step of inputting an audio signal corresponding to the at least one frame and the at least one image frame into a learned neural network model to obtain at least one of whether the special effect is provided, the image information, or the audio information corresponding to the special effect. The learned neural network model may be trained to obtain at least one of whether the special effect is provided, the image information, or the audio information by learning changes in the size and color of the audio signal.

[0024] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0025] FIG. 1A is a diagram schematically illustrating an electronic device according to one or more embodiments of the present disclosure.

[0026] FIG. 1B is a diagram schematically illustrating an electronic device according to one or more embodiments of the present disclosure.

[0027] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0028] FIG. 3 is a block diagram illustrating a detailed configuration of an electronic device according to one or more embodiments.

[0029] FIG. 4 is a flowchart illustrating an image output method of an electronic device according to one or more embodiments.

[0030] FIG. 5A is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0031] FIG. 5b is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0032] FIG. 5c is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0033] FIG. 5d is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0034] FIG. 6 is a flowchart illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0035] FIG. 7A is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0036] FIG. 7b is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0037] FIG. 7c is a diagram illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0038] FIG. 8A is a diagram for identifying maintenance and termination of provision of special effects according to one or more embodiments.

[0039] FIG. 8B is a diagram for identifying maintenance and termination of provision of special effects according to one or more embodiments.

[0040] FIG. 8c is a diagram for identifying maintenance and termination of provision of special effects according to one or more embodiments.

[0041] FIG. 8d is a diagram for identifying maintenance and termination of provision of special effects according to one or more embodiments.

[0042] FIG. 9 is a flowchart illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0043] FIG. 10 is a drawing for explaining an image output method according to one or more embodiments.

[0044] FIG. 11A is a drawing illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0045] FIG. 11b is a drawing illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0046] FIG. 12a is a drawing illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0047] FIG. 12b is a drawing illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0048] FIG. 13 is a flowchart illustrating a special effect emphasis processing method using a picture frame according to one or more embodiments.

[0049] FIG. 14 is a drawing for explaining a special effect emphasis processing method using a picture frame according to one or more embodiments.

[0050] FIG. 15 is a drawing for explaining a special effect emphasis processing method using a picture frame according to one or more embodiments.

[0051] FIG. 16 is a flowchart illustrating a method for identifying whether to provide special effects based on user input according to one or more embodiments.

[0052] FIG. 17 is a flowchart illustrating a method for emphasizing audio special effects according to one or more embodiments.

[0053] FIG. 18 is a flowchart illustrating a method for identifying whether a special effect is provided using a neural network model according to one or more embodiments.

[0054] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0055] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0056] The terminology used in this disclosure has been selected from widely used, current terms, taking into account the functions of the disclosure. However, this may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0057] Terms like "first" and "second" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0058] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0059] "At least one of A and B" and "at least one of A or B" should be interpreted to mean either "A" or "B" or "A and B." As another example, "performing at least one of Step 1 and Step 2" or "performing at least one of Step 1 or Step 2" means the following three juxtapositions: (1) performing Step 1; (2) performing Step 2; and (3) performing Steps 1 and 2.

[0060] Elements described as "modules" or "parts" may be physically implemented by analog and / or digital circuits including one or more of logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, etc.

[0061] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0062] One or more embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings.

[0063] FIGS. 1A and 1B are drawings schematically illustrating an electronic device according to one or more embodiments of the present disclosure.

[0064] According to one or more examples illustrated in FIG. 1a, the electronic device (100) can be implemented as various devices having a display function, such as a monitor, a smart monitor, a smart TV, an electronic picture frame, an electronic blackboard, an electronic table, a laptop, a digital signage, a digital information display (DID), a video wall, a projector, a tablet PC, etc.

[0065] According to one or more examples, the electronic device (100) may communicate with an input device (200). The input device (200) may be implemented as a device such as a keyboard as illustrated, as well as a joystick, a button input device, a wheel input device, a touch pad, or a mouse. However, the input device (200) is not limited thereto, and the input device (200) may also be implemented as a touch screen provided in the electronic device (100) capable of performing both a display function and an operation input function.

[0066] According to one or more examples illustrated in FIG. 1B, the electronic device (100) may be a device pre-registered with the server (300). For example, the server (300) may manage information regarding content provided by other electronic devices as well as the electronic device (100). For example, the electronic device (100) may transmit information related to special effects within the content described below to the server (300), and the server (300) may store and manage the received information. For example, the server (300) may transmit information related to providing special effects to other electronic devices based on the stored information. The server (300) may be implemented as a cloud server, but is not limited thereto.

[0067] Hereinafter, various embodiments of identifying special effect information within content provided by an electronic device (100) and providing special effects to enhance immersiveness and / or three-dimensionality based on the identified information will be described.

[0068] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0069] According to FIG. 2, the electronic device (100) includes a display (110), a memory (120), and at least one processor (130).

[0070] (At least one) display (110) can be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (110) may also include a driving circuit, a backlight unit, etc., which can be implemented in a form such as an a-si (amorphous silicon) TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to one or more examples, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (110) so as to be implemented so as to detect various types of touch inputs. For example, the display (110) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to one or more examples, the display (110) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc.

[0071] (At least one) memory (120) can store data required for various embodiments. The memory (120) may be implemented in the form of a memory embedded in the electronic device (100) or in the form of a memory that can be attached or detached from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100'), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0072] One or more processors (130) control the overall operation of the electronic device (100). Specifically, one or more processors (130) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (130) may be electrically connected to the display (110) and the memory (120) to control the overall operation of the electronic device (100). The processor (130) may be composed of one or more processors.

[0073] One or more processors (130) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (120).

[0074] The one or more processors (130) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (130) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (130) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0075] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0076] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (130) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0077] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0078] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of explanation, one or more processors (130) will be referred to as a processor (130).

[0079] FIG. 3 is a block diagram illustrating a detailed configuration of an electronic device according to one or more embodiments.

[0080] According to FIG. 3, the electronic device (100') may include a display (110), a memory (120), one or more processors (130), a communication interface (140), a user interface (150), a camera (160), a speaker (170), and a sensor (180).

[0081] It goes without saying that the communication interface (140) can be implemented as various interfaces according to one or more implementation examples of the electronic device (100'). For example, the communication interface (140) can communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0082] The user interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0083] The camera (160) can be turned on and take pictures according to a preset event. The camera (160) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then signal processed. For example, the camera (120) can be implemented as a general camera, a stereo camera, a depth camera, etc.

[0084] The speaker (170) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (130) may control the speaker (170) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0085] The sensor (180) may include various types of sensors such as a touch sensor, a proximity sensor, an acceleration sensor (or a gravity sensor), a geomagnetic sensor, a gyro sensor, a pressure sensor, a position sensor, a distance sensor, a light sensor, etc.

[0086] In addition, the electronic device (100') may include a microphone, a tuner, a demodulator, etc. according to one or more examples.

[0087] A microphone is a device configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100') may receive user voice input via an external device through a communication interface (140).

[0088] The tuner can receive RF broadcast signals by tuning to a channel selected by the user or all previously stored channels among RF (Radio Frequency) broadcast signals received through an antenna.

[0089] The demodulator receives and demodulates a digital IF signal (DIF) converted from the tuner, and can also perform channel decoding, etc.

[0090] FIG. 4 is a flowchart illustrating an image output method of an electronic device according to one or more embodiments.

[0091] According to one or more embodiments, the processor (130) can identify whether the size of an audio signal corresponding to at least one image frame included in the content is greater than or equal to a threshold value (S410).

[0092] If the processor (130) identifies that the size of an audio signal corresponding to at least one image frame included in the content is greater than or equal to a threshold value (S410:Y), it can identify whether to provide a special effect based on pixel information included in at least one image frame (S420).

[0093] Here, special effects (also abbreviated as SFX, SPFX, or FX) are effects used in the film, television, theater, video game, and simulator industries to simulate imagined events in stories and virtual worlds. For example, special effects may include visual special effects and / or auditory special effects. For example, visual special effects may include any type of digital technique based on computer graphics (CG).

[0094] In one or more examples, the threshold for identifying whether a special effect is provided may be the average audio signal level of the content. For example, the threshold may be determined by the average audio signal level played to date. In this case, the threshold may be continuously updated over time. In another example, the threshold for identifying whether a special effect is provided may be included in the content's metadata, etc., as a default value.

[0095] According to one or more examples, when an audio signal having a size greater than a threshold value is identified, the processor (130) may analyze pixel information of at least one image frame corresponding to the identified time point and determine whether to provide a special effect based on the analysis result.

[0096] If the processor (130) identifies that a special effect is provided in at least one image frame (S320:Y), the processor (130) can identify image information corresponding to the special effect in at least one image frame (S430). According to one or more examples, the processor (130) can identify pixel values ​​and pixel positions corresponding to the identified special effect in the image frame as image information corresponding to the special effect.

[0097] The processor (130) may perform special effect emphasis processing by outputting a special effect image (or special effect graphic image) acquired based on the identified image information together with at least one image frame (S440). According to one or more examples, the processor (130) may output the special effect image frame and the original image frame to different display layers based on pixel values ​​and pixel positions corresponding to the special effect. For example, the original image frame may be output to a first display layer, and the special effect image may be output to a second display layer located behind the first display layer.

[0098] FIGS. 5A to 5D are drawings for explaining a method for identifying whether a special effect is provided according to one or more embodiments.

[0099] According to one or more embodiments, the processor (130) may identify feature points associated with the occurrence of special effects in an audio signal of content in various ways. As one or more examples, if a feature point predicting the occurrence of a special effect is identified in the audio signal, the processor (130) may determine whether a special effect is provided in an image frame. As another example, the processor (130) may reduce delay by predicting feature points in advance using parameters acquired during the audio decoding process.

[0100] According to one or more embodiments, the processor (130) may identify whether an audio signal having a size greater than or equal to a threshold value (T) is identified in the audio signal of the content, as illustrated in FIG. 5A. For example, in FIG. 5A, it may be determined that the size of the audio signal at point A is less than the threshold value, but the size of the audio signal at points B, C, and D is identified as being greater than or equal to the threshold value.

[0101] Fig. 5b may represent a state in which no special effects are provided for an image frame (510) corresponding to a time point A where the size of the audio signal is less than the threshold value (T). According to one or more examples, the processor (130) may not determine whether to provide special effects for the image frame illustrated in Fig. 5b because the size of the corresponding audio signal is less than the threshold value (T).

[0102] Fig. 5c may represent a state in which a missile special effect is provided in an image frame (520) corresponding to a time point B in which the size of the audio signal is greater than or equal to a threshold value (T). According to one or more examples, the processor (130) may determine whether to provide a special effect in the corresponding image frame, since the size of the corresponding audio signal is greater than or equal to the threshold value (T) in the case of the image frame illustrated in Fig. 5c.

[0103] Fig. 5d may represent a state in which a flame special effect is provided to an image frame (530) corresponding to a point in time D in which the size of the audio signal is greater than or equal to a threshold value (T). According to one or more examples, the processor (130) may determine whether to provide a special effect in the corresponding image frame because the size of the corresponding audio signal is greater than or equal to the threshold value (T) in the case of the image frame illustrated in Fig. 5d.

[0104] According to one or more embodiments, when an audio signal whose size has suddenly increased by more than a preset size compared to the average size of a recent preset time interval is identified in an audio signal of the content, the processor (130) may determine whether to provide a special effect in a corresponding image frame. For example, when an audio signal whose size has suddenly increased by more than 10 DB compared to the average size (DB) for the recent 1 second is identified, the processor (130) may determine whether to provide a special effect in a corresponding image frame.

[0105] According to one or more embodiments, the processor (130) may identify whether to provide a special effect in a corresponding image frame when an audio signal of a preset type is identified in an audio signal of the content.

[0106] For example, if a pre-stored sound related to a special effect is identified in the audio signal of the content, the processor (130) can determine whether the special effect is provided in the corresponding image frame. For example, if a pre-stored sound related to a special effect, such as a missile, gun, sword, scream, thunder, etc., is identified, the processor (130) can determine whether the special effect is provided in the corresponding image frame.

[0107] According to one or more embodiments, the processor (130) may determine whether to provide a special effect in a corresponding image frame when the directionality of sound is identified in the audio signal of the content. For example, the processor (130) may determine whether to provide a special effect in a corresponding image frame when the directionality of sound, such as left to right, up to down, etc., is identified.

[0108] When a feature point of an audio signal is detected according to the various methods described above, the processor (130) can detect a feature point of a video signal. That is, when provision of a special effect is predicted in an audio signal, the processor (130) can identify whether a special effect is provided in a corresponding image frame.

[0109] According to one or more examples, the processor (130) may identify a feature indicating provision of a special effect based on at least one of a change in a color histogram, a size of an area having a saturation difference of x or more compared to a surrounding area, and a duration of time for which the area is maintained. For example, the processor (130) may identify that a special effect is provided if an area having a saturation difference of x compared to a surrounding area is x pixels or 1 / n of the entire screen size or more, and the area is maintained for 0.5 seconds or more.

[0110] Meanwhile, the processor (130) can also reduce delay by predicting whether to provide special effects using parameters obtained during the decoding process of compressed images of a specific compression method (e.g., MPEG / H.264, etc.).

[0111] FIG. 6 is a flowchart illustrating a method for identifying whether a special effect is provided according to one or more embodiments.

[0112] According to one or more embodiments, the processor (130) may obtain color histogram information based on pixel information included in at least one image frame (S610). The color histogram is information representing the distribution of brightness values ​​for pixels within an image. In the color histogram, the horizontal axis represents the brightness values ​​of a 256-gray-level image, and the vertical axis represents the frequency of each brightness value (level), and a higher frequency may be represented as it goes up. Such a histogram intuitively represents whether the brightness values ​​are evenly distributed or biased to one side, so that it is easy to identify whether a special effect occurs.

[0113] The processor (130) can compare the acquired color histogram information with reference color histogram information to obtain histogram difference information (S620). According to one or more examples, the reference color histogram information may be the average color histogram of the corresponding content. For example, the reference color histogram information may be the average histogram value of the video frames played to date. In this case, the reference color histogram information may be continuously updated over time.

[0114] The processor (130) can identify whether a special effect is provided and at least one of image information corresponding to the special effect based on the acquired difference information (S630). According to one or more examples, the processor (130) can identify that a special effect is provided if the acquired difference information is greater than or equal to a threshold value. This is because, since the color corresponding to the special effect is provided in a prominent manner, the histogram difference value can be identified as being greater than or equal to the threshold value when the special effect is provided. Here, the threshold value can be set to different values ​​depending on the type of content (e.g., game content, movie content, etc.) and the detailed type of content (e.g., role-playing game, shooting game, simulation game, action game, puzzle game, adventure game, etc.). The threshold value can be preset or can be set or updated based on the playback frame.

[0115] FIGS. 7A to 7C are drawings for explaining a method for identifying whether a special effect is provided according to one or more embodiments.

[0116] According to one or more embodiments, the processor (130) may obtain a color histogram (511) of an image frame (510) as illustrated in FIG. 7A. In this case, the processor (130) may compare the obtained color histogram (511) with a reference histogram to obtain a difference value, and if the difference value is less than a first threshold value, determine that no special effect is provided. Alternatively, the processor (130) may compare the obtained color histogram (511) with a reference histogram to obtain a similarity value, and if the similarity value is greater than or equal to a second threshold value, determine that no special effect is provided.

[0117] Next, the processor (130) can obtain a color histogram (521) of a subsequent image frame (520) as illustrated in FIG. 7B. In this case, the processor (130) can compare the obtained color histogram (521) with a reference histogram to obtain a difference value, and if the difference value is greater than or equal to a first threshold value, identify that a special effect is provided. Alternatively, the processor (130) can compare the obtained color histogram (521) with a reference histogram to obtain a similarity value, and if the similarity value is less than a second threshold value, identify that a special effect is provided.

[0118] Next, the processor (130) can obtain a color histogram (531) of a subsequent image frame (530) as illustrated in FIG. 7C. In this case, the processor (130) can compare the obtained color histogram (531) with a reference histogram to obtain a difference value, and if the difference value is greater than or equal to a first threshold value, identify that a special effect is provided. Alternatively, the processor (130) can compare the obtained color histogram (531) with a reference histogram to obtain a similarity value, and if the similarity value is less than a second threshold value, identify that a special effect is provided.

[0119] According to one or more examples, the first threshold value and / or the second threshold value in FIGS. 7B and 7C may be the same value as the first threshold value and / or the second threshold value described in FIG. 7A. For example, the first threshold value and / or the second threshold value may be set to a default value within the same content. However, according to another example, the first threshold value and / or the second threshold value may be a different value than the first threshold value and / or the second threshold value described in FIG. 7A. For example, within the same content, the first threshold value and / or the second threshold value may be set to an average value of the image frames played so far (e.g., an average value of the image frames for which no special effects are provided) and may be updated as the number of image frames played increases.

[0120] Meanwhile, if the processor (130) identifies that a special effect is provided in an image frame, the processor (130) may obtain special effect image information based on a color area corresponding to the special effect. According to one or more examples, the processor (130) may crop a color area corresponding to the special effect to generate a special effect image frame. According to another example, the processor (130) may generate a special effect image frame based on coordinate information of a color pixel corresponding to the special effect. According to another example, the processor (130) may use a pre-stored special effect image frame. For example, image frames related to special effects such as missiles, guns, swords, screams, thunder, etc. may be pre-stored in the memory (120), and the processor (130) may use the pre-stored special effect image frame.

[0121] According to one or more examples, the processor (130) may provide a final output image based on the original image frame (or the frame including the reduced-size image area) and the special effect image frame. For example, the processor (130) may provide the final output image by outputting the final output image through different display layers based on the original image frame (or the frame including the reduced-size image area) and the special effect image frame. That is, at least one processor (130) may control the display to output the special effect image to the remaining area of ​​the display excluding the area where the original image frame is displayed, when the original image frame is displayed in an area smaller than the entire area of ​​the display.

[0122] In another example, the processor (130) may obtain a final output image by combining special effect image information with an original image frame (or a frame including a reduced-size image area). For example, the processor (130) may replace pixel values ​​of an area corresponding to a special effect in the original image frame (or a frame including a reduced-size image area) with updated pixel values ​​based on the special effect image information to provide a final output image.

[0123] According to one or more examples, the processor (130) may obtain an output image including a special effect image in a border area outside the image area from a frame including a reduced-size image area. Here, the frame including the reduced-size image area may be a frame in which pixels in the area outside the image area, i.e., the border area, do not have a specific pixel value.

[0124] For example, if a reduced-size image area is enlarged to the screen size of the display (110), a special effect image corresponding to the border area can be obtained, so that an output image including the reduced-size image area and the special effect image corresponding to the border area can be obtained. For example, after an image area that is reduced in size is enlarged to the screen size of the display (110), a special effect image corresponding to the border area can be cropped from the enlarged image, thereby obtaining a special effect image corresponding to the border area.

[0125] According to another example, the processor (130) may provide an output image in a form in which pixel values ​​of a specific area are emphasized in the original image frame (i.e., an image frame whose size has not been reduced). The specific area may include a border area (or an edge area), a center area, etc. The border area may include at least one of the upper / lower / left / right border areas. Here, emphasizing the pixel values ​​may be an adjustment in the form of increasing the luminance value, saturation value, etc. of the pixel values ​​of the border area.

[0126] According to one or more embodiments, the processor (130) may identify whether to maintain or terminate special effect emphasis processing in at least one subsequent frame based on a difference between pixel information included in at least one image frame and pixel information included in at least one subsequent frame.

[0127] Next, the processor (130) can identify a frame section in which special effect emphasis processing is provided based on whether the identified special effect is maintained or terminated. For example, when a frame in which a special effect ends is identified, the processor (130) can identify a frame in which special effect emphasis processing ends based on the frame in which the special effect ends, based on the level of the special effect, the type of the special effect, etc. For example, the processor (130) can determine a frame in which special effect emphasis processing ends based on a playback point of a threshold number of frames before and / or a playback point of a threshold number of frames after the frame in which the special effect ends.

[0128] According to one or more examples, the processor (130) may maintain special effect emphasis processing when a difference between pixel information included in at least one image frame and pixel information included in at least one subsequent frame is less than a preset value. Furthermore, the processor (130) may terminate special effect emphasis processing when a difference between pixel information included in at least one image frame and pixel information included in at least one subsequent frame is greater than or equal to a preset value. For example, the processor (130) may maintain special effect emphasis processing when a difference between an average value of an image frame belonging to a time section in which special effect emphasis processing is provided and an average value of a subsequent frame is less than a preset value, and may terminate special effect emphasis processing when the difference is greater than or equal to the preset value.

[0129] According to one or more embodiments, when the processor (130) determines that the special effect emphasis processing is to be terminated, the processor (130) may terminate the special effect emphasis processing in a fade-out manner. For example, the processor (130) may terminate the special effect emphasis processing in a manner that gradually reduces the special effect emphasis processing level over a plurality of frames.

[0130] FIGS. 8A to 8D are drawings for identifying maintenance and termination of provision of special effects according to one or more embodiments.

[0131] According to one or more embodiments, the processor (130) may maintain a special effect emphasis state even if the size of the audio signal falls below the threshold value (T) during a time period in which the color histogram of the image frame satisfies the special effect provision situation. For example, as illustrated in FIG. 8A, in a first time period (810) in which the color histogram of the image frame satisfies the special effect provision situation, the special effect emphasis state may be maintained even if the size of the audio signal falls below the threshold value (T). For example, in FIG. 8A, it can be confirmed that the size of the audio signal at time A is below the threshold value, but the sizes of the audio signals at time B, time C, and time D are identified as being above the threshold value.

[0132] FIG. 8b may represent a state in which no special effect is provided in an image frame (510) corresponding to a time point A when the size of the audio signal is less than a threshold value (T), and FIG. 8c may represent a state in which a missile special effect is provided in an image frame (520) corresponding to a time point B when the size of the audio signal is greater than or equal to the threshold value (T). In this case, if special effect emphasis processing is started at time point B and the color histogram of the image frame satisfies the special effect provision situation in the first time section (810), the special effect emphasis state can be maintained even if the size of the audio signal falls below the threshold value (T).

[0133] Next, if the image corresponding to the time point C where the audio signal size is greater than or equal to the threshold value (T) is an image for which no special effects are provided, as illustrated in FIG. 8B, the special effect emphasis processing may be terminated. For example, the processor (130) may terminate the special effect emphasis processing if the color histogram of the image frame becomes relatively similar to the color histogram of a frame for which no special effects are provided.

[0134] FIG. 8D may represent a state in which a flame special effect is provided to an image frame (530) corresponding to a time point D in which the size of the audio signal is greater than or equal to a threshold value (T). In this case, if special effect emphasis processing is started at time point D and the color histogram of the image frame satisfies the special effect provision situation in the second time section (820), the special effect emphasis state can be maintained even if the size of the audio signal falls below the threshold value (T).

[0135] FIG. 9 is a flowchart illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0136] According to one or more embodiments, the processor (130) may reduce the size of at least one image frame and provide it according to a preset event (S910).

[0137] The processor (130) can control the display (110) to output a special effect image to the remaining area of ​​the entire display area (excluding the area where the reduced image is displayed).

[0138] FIG. 10 is a drawing for explaining an image output method according to one or more embodiments.

[0139] According to one or more embodiments, the screen size may be adjusted and provided based on a preset event. According to one or more embodiments, when game content is provided on a large screen, the screen may be too large, reducing the user's immersion. In this case, the screen size may be adjusted and provided based on a user command or the user's location.

[0140] In one or more examples, the screen may be provided in a curved form to provide a three-dimensional effect, or key information may be provided in an enlarged form. For example, as illustrated in FIG. 10, an immersive screen with applied curvature, such as a curved screen, may be provided. Furthermore, as illustrated in FIG. 10, distortion caused by the curved effect may be corrected as key information, such as a minimap, is provided in an enlarged form.

[0141] When the screen size is adjusted and provided in this way, blank areas (e.g., border areas) may appear within the screen. However, this is not limited to this, and blank areas such as letter boxes (or black bars) may also appear depending on the difference between the screen's width / height ratio and the content's width / height ratio.

[0142] FIGS. 11A and 11B are drawings illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0143] According to one or more embodiments, the processor (130) can obtain a special effect image based on a specific color included in an image frame (1110).

[0144] For example, when a special effect is provided in response to a missile being launched in an image frame (1110) as illustrated in FIG. 11A, the processor (130) can identify a color corresponding to the special effect and obtain a special effect image (1120). For example, a chroma key synthesis algorithm utilized in broadcast synthesis, etc. may be used as a method of extracting a special effect image based on the identified color, but is not limited thereto.

[0145] According to one or more examples, if the image frame (1110) is a resized image frame (e.g., resized to a curved screen shape), the special effect image may also be in a resized form. Accordingly, the processor (130) may enlarge and scale the size of the acquired special effect image to correspond to the screen size of the display (110). For example, the processor (130) may enlarge the special effect image until the special effect image becomes the same size as the screen size, i.e., the output frame, based on the center of the frame including the reduced image.

[0146] The processor (130) can provide an output image (1130) by providing a second display layer including a special effect image (1121) behind a first display layer including a reduced image (1111).

[0147] According to one or more examples, the processor (130) may obtain an output image by alpha blending an image including a reduced image area and a special effect image. Here, alpha blending refers to a method of displaying a background RGB value and an RGB value above it by assigning a new value called A (Alpha) to the color values ​​RGB to create a transparent effect when overlapping another image on top of the image.

[0148] For example, the Alpha value can be distinguished from 0 to 255 or 0.0 to 1.0, where 0 can mean completely transparent, and the opposite, 255 (or the highest value such as 1.0), can mean fully opaque. Or, 0 can mean completely opaque, and the opposite, 255 (or the highest value such as 1.0) can mean fully transparent. For example, if 8 bits are allocated to the Alpha value and can express values ​​from 0 to 255, the larger the value, the higher the ratio of the corresponding pixel, and the lower the value, the lower the ratio. In one or more examples, when mixing an image (I1) including a reduced image area and a special effect image (I2), the mixing operation can be expressed by a formula such as I1*Alpha + I2*(1-Alpah) or I1*(1-Alapha)+I2*alpha or I1*Alpah+I2. For example, in an image containing a reduced image area, increasing the transparency of the remaining area excluding the image area allows a special effect image provided later to be displayed in that area.

[0149] In one or more examples, since the special effect image is obtained from a reduced image and is therefore smaller than the screen size of the display (110), enlargement scaling is used. Accordingly, when mixing each image using alpha blending, the alpha value corresponding to the special effect image may be upscaled and then mixing may be performed using the upscaled alpha value. For example, at least one scaling technique (or interpolation technique) among polyphase scaling (or polyphase interpolation), trilinear scaling (or trilinear interpolation), linear scaling (or linear interpolation), or bilinear scaling (or bilinear scaling) may be used for scaling the alpha value.

[0150] FIGS. 12A and 12B are drawings illustrating a method for emphasizing special effects according to image reduction according to one or more embodiments.

[0151] According to one or more embodiments, the processor (130) can obtain a special effect image based on a plurality of specific colors included in the image frame (1210).

[0152] For example, when a special effect according to a flame is provided in an image frame (1210) as illustrated in FIG. 12a, the processor (130) can identify a plurality of colors corresponding to the special effect and obtain a special effect image (1220).

[0153] According to one or more examples, if the image frame (1210) is a resized image frame (e.g., resized to a curved screen shape), the special effect image may also be in a resized form. Accordingly, the processor (130) may enlarge and scale the size of the acquired special effect image to correspond to the screen size of the display (110). For example, the processor (130) may enlarge the special effect image until the special effect image becomes the same size as the screen size, i.e., the output frame, based on the center of the frame including the reduced image.

[0154] The processor (130) can provide an output image (1230) by providing a second display layer including a special effect image (1221) behind a first display layer including a reduced image (1211).

[0155] FIG. 13 is a flowchart illustrating a special effect emphasis processing method using frame graphics according to one or more embodiments.

[0156] According to one or more embodiments, the processor (130) may provide a third display layer including a frame graphic on a front end of a first display layer including at least one image frame (S1310).

[0157] The processor (130) can remove a background image outside the frame graphic from at least one image frame (S1320).

[0158] The processor (130) can provide a second display layer including a special effect image to the third display layer front end (S1330).

[0159] FIG. 14 is a drawing for explaining a special effect emphasis processing method using frame graphics according to one or more embodiments.

[0160] According to one or more embodiments, the processor (130) may obtain a special effect image based on a specific color included in the image frame (1410). For example, as illustrated in FIG. 14, if a special effect corresponding to a monster hand is provided in the image frame (1410), the processor (130) may identify a color corresponding to the special effect and obtain a special effect image including the monster hand.

[0161] Additionally, the processor (130) can remove a background image outside the virtual frame graphic (1421) from the image frame (1410) while providing the virtual frame graphic (1421).

[0162] Next, the processor (130) can provide an immersive video (1430) by providing a frame including a special effect image, a frame including a frame graphic, and a video frame with a background image removed as a front layer, a middle layer, and a back layer, respectively.

[0163] FIG. 15 is a drawing for explaining a special effect emphasis processing method using frame graphics according to one or more embodiments.

[0164] According to one or more embodiments, the processor (130) may obtain a special effect image based on a plurality of specific colors included in the image frame (1510). For example, when a special effect corresponding to a gun is provided in the image frame (1510) as illustrated in FIG. 15, the processor (130) may identify a plurality of colors corresponding to the special effect to obtain a special effect image including the gun.

[0165] Additionally, the processor (130) can remove a background image outside the virtual frame graphic (1521) from the image frame (1510) while providing the virtual frame graphic (1521).

[0166] Next, the processor (130) can provide an immersive image (1530) by providing a frame including a special effect image, a frame including a frame graphic, and an image frame with a background image removed as a front layer, a middle layer, and a back layer, respectively.

[0167] FIG. 16 is a flowchart illustrating a method for identifying whether to provide special effects based on user input according to one or more embodiments.

[0168] According to one or more embodiments, when a user input is received (S1610:Y), the processor (130) can identify whether the received user input is user input related to providing a special effect (S1620). According to one or more examples, a user input of a preset type can be identified as user input related to providing a special effect. For example, a user input using a weapon such as a missile, gun, or knife in game content can be identified as user input related to providing a special effect.

[0169] Next, if the received user input is identified as a user input related to providing a special effect (S1620:Y), the processor (130) may determine whether to provide a special effect in at least one image frame based on the received user input (S1630). According to one or more examples, if the user input related to providing a special effect is identified, the processor (130) may determine whether to provide a special effect in the image frame. That is, even if the size of the audio signal is not greater than a threshold value, the processor (130) may determine whether to provide a special effect in the image frame if the user input related to providing a special effect is identified. According to another example, if the user input related to providing a special effect is identified, it is also possible to identify the type of the special effect and immediately identify a color corresponding to the special effect of the corresponding type in the image frame. For example, if the processor (130) identifies a user input related to launching a missile, the processor (130) may identify a color corresponding to a special effect related to launching a missile based on pre-stored information, and obtain a special effect image corresponding to the identified color in the image frame.

[0170] According to one or more embodiments, the sensitivity for determining whether to provide a special effect, the strength of special effect emphasis, etc. may be set based on user input. For example, the threshold value of an audio signal for determining whether to provide a special effect, the threshold value of a color histogram, etc. may be set / changed based on user input. For example, the degree of pixel value increase for special effect emphasis, the size of the special effect emphasis area, etc. may be set / changed based on user input.

[0171] FIG. 17 is a flowchart illustrating a method for emphasizing audio special effects according to one or more embodiments.

[0172] According to one or more embodiments, when it is identified that a special effect is provided in at least one image frame (S1710), the processor (130) may identify audio information corresponding to the special effect in at least one image frame (S1720).

[0173] The processor (130) may output special effect audio acquired based on the identified audio information through the speaker (170) (S1730). Here, the special effect audio may be audio identified in the audio signal or pre-stored audio. For example, audio effects corresponding to special effect types (e.g., missiles, guns, etc.) may be pre-stored.

[0174] According to one or more examples, the processor (130) may output special effect audio so that it is emphasized compared to other audio. For example, the processor (130) may output special effect audio louder than other audio, or output it so that it is emphasized compared to other audio through sound processing such as equalizer processing.

[0175] According to one or more embodiments, special effect audio may be provided along with the emphasis processing of the special effect image. However, this is not limited to the above, and it is of course possible to provide only the special effect audio without the emphasis processing of the special effect image.

[0176] FIG. 18 is a flowchart illustrating a method for identifying whether a special effect is provided using a neural network model according to one or more embodiments.

[0177] According to one or more embodiments, the processor (130) inputs an audio signal corresponding to at least one frame and at least one image frame to at least one learned neural network model (S1810), thereby obtaining at least one of whether a special effect is provided, image information corresponding to the special effect, or audio information corresponding to the special effect (S1820).

[0178] According to one or more examples, at least one learned neural network model can be trained to learn changes in magnitude and color of an audio signal to obtain at least one of whether a special effect is provided, image information corresponding to the special effect, or audio information corresponding to the special effect.

[0179] As one or more examples, at least one neural network model may include a plurality of neural network models trained to output whether a special effect is provided and image information corresponding to the special effect, respectively. As another example, at least one neural network model may include a plurality of neural network models trained to output whether a special effect is provided, image information corresponding to the special effect, and audio information corresponding to the special effect, respectively. As another example, at least one neural network model may include a neural network model trained to output whether a special effect is provided and a neural network model trained to output image information and audio information corresponding to the special effect, respectively. As yet another example, at least one neural network model may include a neural network model trained to output whether a special effect is provided and image information corresponding to the special effect, and a neural network model trained to output audio information corresponding to the special effect.

[0180] Here, learning means that a predefined operation rule or neural network model of desired characteristics is created by applying a learning algorithm to a plurality of learning data. This learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server / system. The neural network model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs the operation of the layer through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0181] A learning algorithm is a method for training a given target device using a large amount of learning data, thereby enabling the given target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The learning algorithms in this disclosure are not limited to the aforementioned examples, unless otherwise specified. For example, a neural network model may be trained based on a database containing content identification information, colors corresponding to special effects, and sounds corresponding to special effects. The data stored in the database may be data collected from terminals of multiple users. For example, a neural network model can be trained using image frames (or / and audio signals) with special effects and labels corresponding to the provision of special effects as input and output learning data, respectively, and image frames (or / and audio signals) with no special effects and labels corresponding to the non-provision of special effects as input and output learning data, respectively. For example, a neural network model can be trained using image frames (or / and audio signals) with special effects and special effect images (or / and special effect sounds) as input and output data, respectively.

[0182] According to one or more embodiments, the electronic device (100) may transmit at least one of image information or audio information corresponding to a special effect identified in at least one image frame to an external device along with information about the content. According to one or more examples, the information about the content may include identification information of the content, information about the playback time at which the special effect is provided, a special effect image, and / or audio information.

[0183] In this case, an external device, such as a content management server, can manage the provision of special effects to other electronic devices based on the received information. Based on the received information (i.e., the special effect provision history), the server can manage the timing of special effect provision for each content, the provided special effect images / sounds, etc., and provide this information to other electronic devices.

[0184] For example, the other electronic device may provide special effect emphasis processing based on information received from the server (e.g., information about the playback point at which the special effect is provided, special effect image and / or audio information).

[0185] According to one or more embodiments, the depth value of an object may be identified based on a depth map corresponding to the content, and multiple objects may be provided on different display layers based on the depth values ​​of the objects to provide an image with increased three-dimensionality. For example, if a special effect is identified in an image frame but it is difficult to identify the special effect image based on color, the special effect image may be obtained using the depth map.

[0186] According to one or more embodiments, at least some of the processing related to providing special effects and obtaining special effect images may be performed on the server. According to one or more embodiments, feature points associated with the occurrence of special effects may be identified on the server.

[0187] For example, the electronic device (100) may transmit a content stream to a server and receive and output video (and / or audio) with special effects enhancement processing from the server. For example, when transmitting decoded video / audio to a server, a transmission burden may be incurred depending on the amount of data, so the content stream before decoding may be transmitted to the server.

[0188] According to one or more embodiments, a content provider may provide information about the timing of providing special effects, the method of providing special effects, the form of providing special effects, etc., through metadata, and in this case, the electronic device (100) may perform special effect emphasis processing based on the information.

[0189] According to the various embodiments described above, content can be provided with enhanced immersion and three-dimensionality by extracting special effects from content and highlighting and presenting them. Furthermore, the user experience can be enhanced by utilizing software graphics calculations without requiring specialized hardware to provide three-dimensionality.

[0190] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0191] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0192] According to one or more embodiments of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When an instruction is executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instruction using other components. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0193] Furthermore, according to one or more embodiments of the present disclosure, the methods according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0194] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0195] While specific embodiments of the present disclosure have been particularly shown and described above, it should be understood that various changes in form and details may be made therein without departing from the spirit and scope of the claims that follow.

Claims

1. In electronic devices, display; At least one memory storing at least one instruction; and At least one processor operatively connected to the display and the at least one memory; The at least one processor, by executing the at least one instruction, If the size of an audio signal corresponding to at least one image frame included in the content is identified as being greater than a threshold value, whether to provide a special effect is identified based on pixel information included in the at least one image frame, If it is identified that a special effect is provided in at least one image frame, image information corresponding to the special effect is identified in the at least one image frame, An electronic device that outputs a special effect image obtained based on the above image information through the display together with the at least one image frame.

2. In paragraph 1, The one or more processors, by executing the at least one instruction, An electronic device that controls the display so that the special effect image is output to an area remaining in the display excluding the area where the at least one image frame is displayed, when the at least one image frame is displayed in an area smaller than the total area of ​​the display.

3. In paragraph 2, The at least one processor, by executing the at least one instruction, Providing a first display layer comprising at least one image frame, An electronic device that controls the display so that a second display layer including the special effect image is provided behind the first display layer.

4. In paragraph 1, The at least one processor, by executing the at least one instruction, Providing a first display layer comprising at least one image frame, Providing a third display layer including a frame graphic on the first display layer, removing a background image outside the frame graphic from at least one image frame; An electronic device that controls the display so that a second display layer including the special effect image is provided on the third display layer front end.

5. In paragraph 1, further including a user interface; The at least one processor, by executing the at least one instruction, When user input is received through the user interface, identifying whether the user input is related to providing the special effect; An electronic device that identifies whether to provide the special effect in the at least one image frame based on the user input, when the user input is identified as being related to providing the special effect.

6. In paragraph 1, The one or more processors, by executing the at least one instruction, Obtain color histogram information based on the above pixel information, By comparing the above color histogram information with the reference color histogram information, histogram difference information is obtained, An electronic device that identifies whether to provide the special effect or at least one of the image information based on the histogram difference information.

7. In paragraph 1, Speaker; including more; The one or more processors, by executing the at least one instruction, If it is identified that a special effect is provided in at least one image frame, audio information corresponding to the special effect is identified in the at least one image frame, An electronic device that outputs special effect audio through the speaker based on the above audio information.

8. In paragraph 1, The one or more processors, by executing the at least one instruction, By inputting the at least one frame and the audio signal corresponding to the at least one image frame to the learned neural network model, at least one of the image information or the audio information corresponding to the special effect is obtained, whether the special effect is provided, The above learned neural network model is, An electronic device, which learns to learn changes in the size and color of an audio signal to provide the special effect, and to obtain at least one of the image information or the audio information.

9. In paragraph 1, The one or more processors, by executing the at least one instruction, Identifying whether the special effect is maintained or terminated in the at least one subsequent frame based on the difference between the pixel information and the pixel information included in the at least one subsequent frame; An electronic device that identifies a frame section providing the special effect image based on whether the identified special effect is maintained or terminated.

10. In paragraph 1, further comprising a communication interface; The one or more processors, by executing the at least one instruction, Transmitting at least one of the video information or audio information corresponding to the special effect to an external device through the communication interface together with information about the content, Information about the above content: An electronic device comprising identification information of the content and information about the playback time at which the special effect is provided.

11. In the method of outputting images from an electronic device, A step of identifying whether to provide a special effect based on pixel information included in at least one image frame, when the size of an audio signal corresponding to at least one image frame included in the content is identified as being greater than a threshold value; If it is identified that a special effect is provided in at least one image frame, a step of identifying image information corresponding to the special effect in the at least one image frame; and An image output method comprising: a step of outputting a special effect image obtained based on the image information together with at least one image frame.

12. In paragraph 11, The step of outputting together with at least one image frame is: An image output method comprising: a step of outputting the special effect image to an area remaining in the display excluding the area in which the at least one image frame is displayed, when the at least one image frame is displayed in an area smaller than the entire area of ​​the display; 13. In paragraph 12, The step of outputting together with at least one image frame is: providing a first display layer comprising at least one image frame; and An image output method comprising the step of providing a second display layer including the special effect image displayed behind the first display layer.

14. In paragraph 11, The step of outputting together with at least one image frame is: A step of providing a first display layer including at least one image frame; A step of providing a third display layer including a frame graphic on the first display layer front end; A step of removing a background image outside the frame graphic from at least one image frame; and An image output method comprising the step of providing a second display layer including the special effect image on the front side of the third display layer.

15. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, The above actions are, A step of identifying whether to provide a special effect based on pixel information included in at least one image frame, when the size of an audio signal corresponding to at least one image frame included in the content is identified as being greater than a threshold value; If it is identified that a special effect is provided in at least one image frame, a step of identifying image information corresponding to the special effect in the at least one image frame; and A non-transitory computer-readable medium, comprising: a step of outputting a special effect image obtained based on the image information together with at least one image frame.

Citation Information

Patent Citations

  • Haptic feedback sensations based on audio output fromcomputer devices

    KR100925293B1

  • Video game system and apparatus and method for transformation of video and audio signal

    KR1020090063675A

  • Display apparatus and controlling method of thereof

    KR1020170064261A

  • Battery short inspection device

    KR102493137B1

  • KR20220137067A