Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device includes: a memory for storing one or more instructions; and one or more processors connected to the memory and controlling the electronic device, wherein the one or more processors execute the one or more instructions to: identify an area in which a motion is maintained in an input image; identify motion information of the identified area and representative motion information of the identified area; identify motion compensation value on the basis of the identified motion information and the identified representative motion information; perform motion compensation on an input frame included in the input image by applying the identified motion compensation value to the identified area; obtain an interpolation frame by applying the identified motion compensation value to the identified area; and obtain an output image on the basis of the motion-compensated input frame and the obtained interpolation frame.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that performs motion compensation for an input image and a control method thereof.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, the development and distribution of display devices such as TVs and mobile devices are rapidly progressing.

[0003] For example, various frame interpolation methods are being studied to provide smoother motion to users in order to provide better quality images.

[0004] An electronic device according to one or more embodiments includes a memory storing one or more commands and one or more processors, wherein the one or more processors execute the one or more commands to identify a region in which motion is maintained in an input image, identify motion information of the identified region and representative motion information of the identified region, identify a motion compensation value based on the identified motion information and the identified representative motion information, perform motion compensation on an input frame included in the input image by applying the identified motion compensation value to the identified region, obtain an interpolated frame by applying the identified motion compensation value to the identified region, and obtain an output image based on the input frame on which motion compensation is performed and the obtained interpolated frame.

[0005] According to one or more embodiments, the one or more processors can obtain an interpolated frame by executing the one or more instructions, using the identified motion compensation value for the identified region and not using the identified motion compensation value for a region other than the identified region.

[0006] According to one or more embodiments, the one or more processors can perform motion compensation on the input frame by executing the one or more instructions, using the identified motion compensation value for the identified region and not using the identified motion compensation value for a region outside the identified region.

[0007] According to one or more embodiments, the one or more processors can perform motion compensation on the identified region based on the identified motion compensation value in a plurality of input frames included in the input image by executing the one or more commands, and obtain an output image based on the obtained interpolated frame and the plurality of input frames on which the motion compensation was performed.

[0008] According to one or more embodiments, the one or more processors can, by executing the one or more commands, obtain a plurality of motion vectors corresponding to the identified region from a plurality of input frames included in the input image, and identify a representative motion vector that is continuously maintained in the identified region based on the obtained plurality of motion vectors.

[0009] According to one or more embodiments, the one or more processors can identify a motion compensation value corresponding to each of the plurality of input frames based on a difference value between each of the plurality of motion vectors corresponding to the identified region and the representative motion vector by executing the one or more instructions.

[0010] According to one or more embodiments, the one or more processors can identify a motion compensation value corresponding to each of the plurality of input frames by applying a preset filter to a difference value between each of the plurality of motion vectors corresponding to the identified area and the representative motion vector by executing the one or more commands.

[0011] According to one or more embodiments, the one or more processors can identify an area where the motion is maintained based on motion vector statistics for each of a plurality of pixel areas included in the input image by executing the one or more instructions, or identify an area where the motion is maintained by inputting the input image to a learned artificial intelligence model.

[0012] According to one or more embodiments, the one or more processors can identify, as a motion vector of the identified region, a movement distance at which the sum of differences in luminance values ​​of pixels included in the identified region among a plurality of input frames included in the input image is minimum by executing the one or more commands.

[0013] According to one or more embodiments, the one or more processors can, by executing the one or more commands, identify pixels included in the identified area into a plurality of pixel areas, and identify a motion vector of the identified area based on statistical values ​​of the motion vectors identified in units of the plurality of pixel areas.

[0014] According to one or more embodiments, the region where the movement is maintained may include an region where text continues to move at a specific location in the input image.

[0015] According to one or more embodiments, a control method of an electronic device includes the steps of: identifying an area in which motion is maintained in an input image; identifying motion information of the identified area and representative motion information of the identified area; identifying a motion compensation value based on the identified motion information and the identified representative motion information; obtaining an output image by applying the identified motion compensation value to the identified area; performing motion compensation on an input frame included in the input image by applying the identified motion compensation value to the identified area; obtaining an interpolated frame by applying the identified motion compensation value to the identified area; and obtaining an output image based on the input frame on which motion compensation is performed and the obtained interpolated frame.

[0016] According to one or more embodiments, a non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation includes: identifying an area in which motion is maintained in an input image; identifying motion information of the identified area and representative motion information of the identified area; identifying a motion compensation value based on the identified motion information and the identified representative motion information; obtaining an output image by applying the identified motion compensation value to the identified area; performing motion compensation on an input frame included in the input image by applying the identified motion compensation value to the identified area; obtaining an interpolated frame by applying the identified motion compensation value to the identified area; and obtaining an output image based on the input frame on which the motion compensation was performed and the obtained interpolated frame.

[0017] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0018] FIG. 1 is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure.

[0019] FIG. 2A is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0020] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0021] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0022] FIG. 4 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0023] FIG. 5 is a drawing for explaining in detail a method for obtaining an output image according to one or more embodiments.

[0024] FIG. 6 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0025] FIG. 7 is a diagram illustrating a Kalman filtering method according to one or more embodiments.

[0026] FIG. 8 is a diagram illustrating a motion compensation method when frame rate conversion is not required according to one or more embodiments.

[0027] FIG. 9 is a diagram illustrating a motion compensation method when frame rate conversion is required according to one or more embodiments.

[0028] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0029] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions or cases of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0030] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0031] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0032] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0033] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0034] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0035] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0036] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0038] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0039] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0040] FIG. 1 is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure.

[0041] According to the example illustrated in FIG. 1, the electronic device (100) can be implemented as various devices having a display function, such as a monitor, a smart monitor, a smart TV, an electronic picture frame, an electronic blackboard, an electronic table, a laptop, a digital signage, a digital information display (DID), a video wall, a projector, a tablet PC, etc.

[0042] The electronic device (100) can receive various compressed images or images of various resolutions. For example, the electronic device (100) can receive images in a compressed form such as MPEG (Moving Picture Experts Group) (e.g., MP2, MP4, MP7, etc.), JPEG (joint photographic coding experts group), AVC (Advanced Video Coding), H.264, H.265, HEVC (High Efficiency Video Codec), etc. Alternatively, the electronic device (100) can receive any one of SD (Standard Definition), HD (High Definition), Full HD, Ultra HD, or higher resolution images.

[0043] For example, the electronic device (100) may perform frame rate conversion when the input frame rate of the input image and the output frame rate of the output image are different. For example, the electronic device (100) may estimate motion by dividing the input image into pixel units or block units, which are groups of one or more pixels, based on the current frame and the previous frame, and may perform frame interpolation to generate a new frame between the current frame and the previous frame using the estimated motion, thereby performing frame rate conversion.

[0044] For example, during the compression or transmission of a video, or during the frame rate conversion process to transmit at a frame rate different from the frame rate during video production, motion freezes, jumps, and speed changes that did not exist in the original video may occur. For example, when simply interpolating between two input frames to create a new frame, even if motion freezes between the two input frames, an interpolated frame is created by assuming that there is no motion, thus creating a freeze frame. In addition, even when a jump occurs between two frames, a large motion value is detected according to the accelerated motion, and an interpolated frame with fast movement is created between the two frames. In this case, motion freezes or speedups that existed in the input video are estimated between the two input videos, and an interpolated frame is created.

[0045] For example, in the case of text moving at a constant speed, mainly at the bottom of a news video as illustrated in Figure 1, the text may briefly stop in the middle of the movement even though it is continuously moving at a constant speed, or may quickly jump and then return to a constant speed when it resumes. Alternatively, as the speed changes slightly, the PPF (Pixel per frame) may change frequently. While this phenomenon is difficult to recognize when text changes from still to different text, this phenomenon is easily observed in a video with constant movement, and this can cause discomfort to the viewer.

[0046] Accordingly, below, various embodiments that provide smooth motion through motion compensation in the motion continuation region will be described.

[0047] FIG. 2A is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0048] According to FIG. 2a, the electronic device (100) includes a memory (110) and one or more processors (120).

[0049] The memory (110) can store data required for various embodiments. Depending on the purpose of data storage, the memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0050] One or more processors (120) control the overall operation of the electronic device (100). Specifically, one or more processors (120) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (120) may be electrically connected to the display (130) and the memory (110) to control the overall operation of the electronic device (100). The processor (120) may be composed of one or more processors.

[0051] One or more processors (120) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (110).

[0052] The one or more processors (120) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (120) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (120) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0053] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0054] One or more processors (120) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (120) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0055] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0056] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of explanation, one or more processors (120) will be referred to as a processor (120).

[0057] According to one embodiment, the processor (120) may identify an area in the input image where motion is maintained. For example, the processor (120) may identify an area in the input image where motion is maintained based on a plurality of input frames constituting the input image. For example, the area where motion is maintained may be an area where the movement of an object continues within a preset motion range across a plurality of input frames. For example, the area where motion is maintained may include an area where text continues to move at a specific location in the input image.

[0058] Next, the processor (120) can identify movement information of the identified area and representative movement information of the identified area. For example, the movement information of the identified area may be information that quantifies the movement of the identified area. For example, the movement information may include a motion vector. For example, when objects included in multiple frames are connected, a line is created, and the movement of the object creates a strong directionality and gives the feeling of speed. The directional force acting within the screen in this way is called a motion vector.

[0059] For example, the representative motion information of the identified region may be statistically quantified information regarding the motion of the identified region. For example, the representative motion information may include a representative motion vector identified based on statistical values ​​of a plurality of motion vectors corresponding to the identified region.

[0060] Next, the processor (120) can identify a motion compensation value based on the identified motion information and the identified representative motion information. For example, the processor (120) can identify a motion compensation value corresponding to each of the plurality of video frames based on the difference value between each of the plurality of motion vectors corresponding to the identified region and the representative motion vector.

[0061] Next, the processor (120) can obtain an output image by applying the identified motion compensation value to the identified region. For example, the processor (120) can obtain an output image by applying the identified motion compensation value to the identified region in each input frame and not applying the identified motion compensation value to regions other than the identified region.

[0062] For example, the processor (120) may not need to generate an interpolation frame when the input frame rate of the input image and the output frame rate of the output image are the same. In this case, the processor (120) may obtain an output image by performing motion compensation on a plurality of input frames included in the input image without generating an interpolation frame. The input frame rate of the input image is a number indicating how many frames the input image has on the screen per second, and the unit may be Hz (Hertz). The output frame rate of the output image is a number indicating how many frames the output image has on the screen per second, and the unit may be Hz (Hertz).

[0063] For example, the processor (120) may obtain an output frame by applying an identified motion compensation value to an area identified in each of a plurality of input frames and not applying the identified motion compensation value to an area other than the identified area. The output frame may be a frame output on the screen of the display (130).

[0064] According to another example, the processor (120) may need to generate an interpolated frame because the input frame rate of the input image and the output frame rate of the output image are different. The processor (120) may obtain an interpolated frame by applying an identified motion compensation value to an identified region in the input image and not applying the identified motion compensation value to an area outside the identified region. For example, the processor (120) may correct an input frame by applying an identified motion compensation value to an identified region in each of a plurality of input frames and not applying the identified motion compensation value to an area outside the identified region, and obtain an output frame including the corrected input frame and the interpolated frame.

[0065] According to one embodiment, the processor (120) may obtain a plurality of motion vectors corresponding to an area identified from a plurality of input frames, and identify a representative motion vector that is continuously maintained in the identified area based on the plurality of motion vectors.

[0066] According to one embodiment, the processor (120) may identify a motion compensation value corresponding to each of the plurality of image frames based on a difference value between each of the plurality of motion vectors corresponding to the identified region and a representative motion vector. For example, the processor (120) may identify a motion compensation value corresponding to each of the plurality of image frames by applying a preset filter to a difference value between each of the plurality of motion vectors corresponding to the identified region and a representative motion vector.

[0067] According to one embodiment, the processor (120) may identify a region where motion is maintained based on motion vector statistics for each of a plurality of pixel regions included in the input image. Alternatively, the processor (120) may input the input image to a trained artificial intelligence model to identify a region where motion is maintained. Alternatively, the processor (120) may identify a region where motion is maintained by combining the identification results of the former and the latter.

[0068] According to one embodiment, the processor (120) can identify a movement distance at which the sum of differences in luminance values ​​of pixels included in an identified area in a plurality of input frames included in an input image is minimum as a motion vector of the identified area.

[0069] According to one embodiment, the processor (120) can identify pixels included in the identified area into a plurality of pixel areas, and identify a motion vector of the identified area based on statistical values ​​of the identified motion vector in units of the identified pixel areas.

[0070] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0071] According to FIG. 2b, the electronic device (100') may include a memory (110), one or more processors (120), a display (130), a communication interface (140), a user interface (150), a camera (160), a speaker (170), and a sensor (180). Among the configurations illustrated in FIG. 2b, a detailed description of configurations that overlap with those illustrated in FIG. 2a will be omitted.

[0072] The display (130) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (130) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to an example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (130) so as to be implemented so as to detect various types of touch inputs. For example, the display (130) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to an example, the display (130) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc. According to an example, the processor (120) can display an acquired output image through the display (130).

[0073] It goes without saying that the communication interface (140) can be implemented as various interfaces depending on the implementation example of the electronic device (100'). For example, the communication interface (140) can communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. According to one example, the communication interface (140) can communicate with another electronic device, an external server, and / or a remote control device. According to an example, the processor (120) can receive an input image through a communication interface (140).

[0074] The user interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0075] The camera (160) can be turned on and take pictures according to a preset event. The camera (160) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then signal processed. For example, the camera (120) can be implemented as a general camera, a stereo camera, a depth camera, etc.

[0076] The speaker (170) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (120) may control the speaker (170) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0077] The sensor (180) may include various types of sensors such as a touch sensor, a proximity sensor, an acceleration sensor (or a gravity sensor), a geomagnetic sensor, a gyro sensor, a pressure sensor, a position sensor, a distance sensor, a light sensor, etc.

[0078] In addition, the electronic device (100') may include a microphone (not shown), a tuner (not shown), and a demodulator (not shown) depending on the implementation example.

[0079] A microphone (not shown) is configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100') may receive user voice input via an external device through a communication interface (140).

[0080] A tuner (not shown) can receive RF broadcast signals by tuning to a channel selected by a user or all previously stored channels among RF (Radio Frequency) broadcast signals received through an antenna.

[0081] The demodulator (not shown) receives and demodulates a digital IF signal (DIF) converted from the tuner, and can also perform channel decoding, etc.

[0082] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0083] According to FIG. 3, the electronic device (100) can identify an area in which motion is maintained in an input image (S310). For example, the area in which motion is maintained may include an area in which text continuously moves at a specific location in the input image. For example, the electronic device (100) can identify an area in which motion is maintained based on motion vector statistics for each of a plurality of pixel areas included in the input image. Alternatively, the electronic device (100) can input the input image into a trained artificial intelligence model to identify an area in which motion is maintained. Alternatively, the electronic device (100) can identify an area in which motion is maintained by combining the identification results of the former and the latter.

[0084] Next, the electronic device (100) can identify motion information of the identified region and representative motion information of the identified region (S320). For example, the electronic device (100) can obtain a plurality of motion vectors corresponding to the identified region from a plurality of input frames included in the input image. For example, the electronic device (100) can identify a movement distance at which the sum of the differences in luminance values ​​of pixels included in the identified region from a plurality of input frames included in the input image is minimum as the motion vector of the identified region. For example, the electronic device (100) can identify pixels included in the identified region as a plurality of pixel regions, and identify the motion vector of the identified region based on statistical values ​​of the motion vectors identified in units of identified pixel regions. Next, the electronic device (100) can identify a representative motion vector that is continuously maintained in the identified region based on the plurality of motion vectors.

[0085] Next, the electronic device (100) can identify a motion compensation value based on the identified motion information and the identified representative motion information (S330). For example, the electronic device (100) can identify a motion compensation value corresponding to each of a plurality of image frames based on a difference value between each of a plurality of motion vectors corresponding to the identified region and the representative motion vector. For example, the electronic device (100) can identify a motion compensation value corresponding to each of the plurality of image frames by applying a preset filter to a difference value between each of a plurality of motion vectors corresponding to the identified region and the representative motion vector.

[0086] Next, the electronic device (100) can obtain an output image by applying the identified motion compensation value to the identified area (S340). For example, the electronic device (100) can obtain an output image by applying the identified motion compensation value to the identified area in each input frame and not applying the identified motion compensation value to areas other than the identified area.

[0087] Meanwhile, in Fig. 3, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0088] FIG. 4 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0089] Among the operations illustrated in Fig. 4, detailed descriptions of operations that overlap with those illustrated in Fig. 3 will be omitted.

[0090] According to FIG. 4, the electronic device (100) can identify an area in which movement is maintained in an input image (S410).

[0091] Next, the electronic device (100) can identify a motion vector of the identified area and a representative motion vector of the identified area (S420).

[0092] Next, the electronic device (100) can identify a motion compensation value based on the identified motion vector and the identified representative motion vector (S430).

[0093] Next, the electronic device (100) can obtain an interpolated frame by applying the identified motion compensation value to the identified area and not applying the identified motion compensation value to an area other than the identified area (S440).

[0094] According to one example, when the input frame rate of an input image and the output frame rate of an output image are different and interpolation frame generation is required, the electronic device (100) can obtain an interpolation frame by applying an identified motion compensation value to an area identified in the input image and not applying the identified motion compensation value to an area other than the identified area.

[0095] Next, the electronic device (100) may perform motion compensation of the identified region based on the motion compensation values ​​identified in the plurality of input frames included in the input image (S450). For example, the electronic device (100) may apply the identified motion compensation values ​​to the identified regions in each of the plurality of input frames and compensate the input frames without applying the identified motion compensation values ​​to regions other than the identified regions.

[0096] Next, the electronic device (100) can obtain an output image based on the acquired interpolated frame and a plurality of input frames on which motion compensation has been performed (S460).

[0097] Meanwhile, in Fig. 4, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0098] FIG. 5 is a drawing for explaining in detail a method for obtaining an output image according to one or more embodiments.

[0099] As illustrated in FIG. 5, the processor (140) can obtain an output image using a motion estimation module (510), a motion sustained region detection module (520), a region-limited motion vector calculation module (530), a region-limited representative motion vector prediction module (540), a correction vector value calculation module (550), a corresponding region motion correction module (560), a motion compensation module (570), a storage module (580), and a mixing module (590). Here, each module can be implemented with at least one software, at least one hardware, and / or a combination thereof.

[0100] For example, at least one of the motion estimation module (510), the motion sustained region detection module (520), the region-limited motion vector calculation module (530), the region-limited representative motion vector prediction module (540), the correction vector value calculation module (550), the region-specific motion compensation module (560), and the motion compensation module (570) may be implemented to use a predefined algorithm, a predefined formula, and / or a learned artificial intelligence model. The motion estimation module (510), the motion sustained region detection module (520), the region-limited motion vector calculation module (530), the region-limited representative motion vector prediction module (540), the correction vector value calculation module (550), the region-specific motion compensation module (560), the motion compensation module (570), the storage module (580), and the mixing module (590) may be included within the electronic device (100), but may be distributed to at least one external device according to an example.

[0101] In one example, the processor (120) may perform motion estimation on an input image using the motion estimation module (510). For example, the processor (120) may perform motion estimation by identifying a motion vector (or motion vector). In one example, the processor (120) may perform motion estimation using a block matching algorithm, but is not necessarily limited thereto.

[0102] For example, the processor (120) can divide an image frame into small blocks, predict the block to which the current block has moved in time, and identify a motion vector corresponding to each block. As an example, the processor (120) can identify a motion vector using forward motion estimation, which estimates the movement of a reference block by searching a search area in a subsequent frame based on a reference block identified in a previous frame. However, in some cases, it may also be possible to use backward motion estimation.

[0103] For example, the processor (120) may store the received input image, the motion estimation value obtained through the motion estimation module (510), etc. in the storage module (580). For example, the storage module (580) may be implemented as an example of the memory (110). For example, the processor (120) may obtain a frame required for motion estimation from the storage module (580).

[0104] For example, the processor (120) can identify a region within an input image where motion is to be maintained continuously using the motion sustained region detection module (520). For example, in a news image, a region where text is to be continuously moved at the bottom or at a specific location can be identified. For example, when a motion vector for each position is calculated from the input image, the processor (120) can analyze the motion vectors calculated from a plurality of input frames to identify a region where motion is to be maintained continuously (hereinafter, “motion sustained region”).

[0105] For example, the processor (120) may identify a motion continuation region based on statistics of motion vectors derived from multiple input frames. For example, the statistics of the motion vectors may include at least one of a representative value, a median value, or an average value of the multiple motion vectors.

[0106] As another example, the processor (120) may identify a motion-sustaining region using an artificial intelligence model. For example, as illustrated in FIG. 6 , the processor (120) may input multiple input frames into the artificial intelligence model to obtain information about the motion-sustaining region. For example, the information about the motion-sustaining region may include location information (or coordinate information) about the region.

[0107] An artificial intelligence model can learn the location information of areas with a constant velocity within an image based on a database containing various types of input images. Creating an artificial intelligence model through learning means that a predefined motion rule or artificial intelligence model with desired characteristics is created by applying a learning algorithm to a plurality of learning data. This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server / system. The artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its calculations based on the calculation results of the previous layer and at least one defined calculation. Examples of neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the examples described above unless otherwise specified. A learning algorithm is a method of training a given target device using a plurality of learning data so that the given target device can make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The learning algorithms in the present disclosure are not limited to the examples described above unless otherwise specified.

[0108] However, the method for identifying the location information of the motion persistence area in the input image is not limited to the above-described example, and various methods may be used depending on the embodiment.

[0109] According to an example, the processor (120) can use the region-limited motion vector calculation module (530) to calculate a motion vector of a motion continuation region.

[0110] For example, the processor (120) can calculate a motion vector of a motion continuous region using two consecutive frames. For example, the movement distance corresponding to the point where the difference in the luminance values ​​of pixels in the two frames is the smallest can be calculated as a motion vector. The luminance values ​​of two consecutive frames are f t (Y), f t-1 (Y) and assuming the motion continuity area to be a rectangle for convenience, all pixel coordinates within the horizontal width are b j , all pixel coordinates in the vertical b i , search_range the vertical range to find movement y , search_range horizontal range x The sum of the differences in luminance values ​​within the area is called MAD. area When it comes to MAD area can be defined as in the following mathematical formula 1.

[0111]

[0112] for 0 ≤ b i ≤ area width, 0 ≤ b j ≤ area height,

[0113] - search_range x < x < search_range x

[0114] - search_range y < y < search_range y

[0115] The x, y values ​​that minimize these values ​​among all ranges within the search range are the motion vector MV(Dx,Dy) of that area. t can be defined as, for example, minMAD t and MV(Dx,Dy) t can be defined as in the following mathematical equations 2 and 3.

[0116]

[0117]

[0118] As another example, the processor (120) may define a corresponding area as a block by grouping one or more pixels, then extract motion vector values ​​in block units and calculate a motion vector based on statistics of the motion vector values. For example, the statistics of the motion vector values ​​may include at least one of a representative value, an average value, or a median value. For example, the size of a pixel group may be preset during the manufacturing of the electronic device (100) or may be set / changed by a user. In addition, the size of a pixel group is not fixed and may change depending on the resolution, type, etc. of the input image.

[0119] However, the method for identifying a motion vector of a motion continuous region in an input image is not limited to the above-described example, and various methods may be used depending on the embodiment.

[0120] According to an example, the processor (120) can use the region-limited representative motion vector prediction module (540) to produce a representative motion vector of a motion continuation region.

[0121] For example, the processor (120) may generate a motion vector MV(Dx,Dy) of a motion continuation area. t We can use it to derive a continuous motion representative vector. For example, MV(Dx,Dy) t is frame f t-1 , f tIt can be a motion vector produced between. The processor (120) MV(Dx,Dy) t is a continuous representative motion vector (or predicted motion vector) Exp_MV(Dx,Dy) that remains constant over time. t can be produced. For example, the processor (120) can produce a representative motion vector through Kalman filtering as in the following mathematical expression 4.

[0122]

[0123] For example, as illustrated in Figure 7, a Kalman filter may be a recursive filter that estimates the next predicted value based on a measured value, and then compensates for the measured value to derive an optimal predicted value. For example, a linear Kalman filter may be used, but is not necessarily limited thereto. Furthermore, other methods, such as histograms and averages, may also be used to derive representative motion vectors.

[0124] According to an example, the processor (120) uses the correction vector value calculation module (550) to generate a representative motion vector Exp_MV(Dx,Dy) in the motion continuation area. t and motion vector MV(Dx,Dy) t The difference value can be calculated. For example, the representative motion vector Exp_MV(Dx,Dy) t and motion vector MV(Dx,Dy) t The difference value can be calculated as shown in the following mathematical formula 5.

[0125]

[0126] The processor (120) can use various functions func() as in Equation 6 below to use the motion difference value calculated through Equation 5 for motion compensation. For example, the processor (120) filters the motion difference value using the IIR (Infinite impulse response) filtering function and converts the resulting value into a motion compensation value Gap(Dx,Dy). t can be used as

[0127]

[0128] According to one example, the processor (120) can compensate for an input frame and / or generate an interpolated frame using the corresponding area motion compensation module (560) and the motion compensation module (570).

[0129] For example, when frame rate conversion is required, the processor (120) may generate an interpolated frame. For example, the processor (120) may use a motion vector calculated through a region-limited motion vector calculation module (530) for a motion-sustaining region, and may generate an interpolated frame based on a motion vector calculated through a motion estimation module (510) for other regions. For example, the processor (120) may generate an interpolated frame based on the same method as the conventional method for a region other than the motion-sustaining region. For example, the processor (120) may generate an interpolated frame based on a motion estimation value obtained through the motion estimation module (510) for a region other than the motion-sustaining region. Through this, a phenomenon in which motion may be locally processed differently in the motion-sustaining region may be eliminated, so that spatially uniform motion may appear.

[0130] In addition, the processor (120) calculates the motion compensation value Gap(Dx,Dy) calculated in mathematical expression 6. tMotion compensation can be performed on the interpolated frame generated based on . For example, if the interpolated frame generated by applying different methods according to the area is fi, motion compensation can be performed at coordinates x and y based on the following mathematical expression 7.

[0131]

[0132] The processor (120) processes frame f at time t-1 t-1 and frame f at time t t In addition to generating interpolated frames between them, motion compensation is performed on the input frame itself in different ways for the motion-continuous region and other regions to generate a new frame f'. t-1 , f' t can be converted to

[0133] In another example, if frame rate conversion is not required, motion compensation is performed differently for the motion-continuous region and other regions of the input frame without generating an interpolated frame, thereby generating a new frame f'. t-1 , f' t can be converted to

[0134] According to an example, the processor (120) can obtain an output image by mixing a corrected input frame and an interpolated frame using a mixing module (590).

[0135] FIG. 8 is a diagram illustrating a motion compensation method when frame rate conversion is not required according to one or more embodiments.

[0136] According to FIG. 8, the processor (120) may perform different motion compensation for each region for a plurality of input frames (810, 820, 830) included in an input image. For example, the processor (120) may perform motion compensation for a motion sustained region (811, 821, 822) identified in each of the plurality of input frames (810, 820, 830) based on a compensation difference value calculated through a region-limited motion vector calculation module (530), a region-limited representative motion vector prediction module (540), and a compensation vector value calculation module (550). In addition, the processor (120) may not perform motion compensation for a region (812, 822, 832) other than the motion sustained region (811, 821, 822) identified in each of the plurality of input frames (810, 820, 830).

[0137] FIG. 9 is a diagram illustrating a motion compensation method when frame rate conversion is required according to one or more embodiments.

[0138] According to FIG. 9, the processor (120) can obtain interpolated frames (910, 920) based on a plurality of input frames (810, 820, 830) included in an input image. For example, the processor (120) can generate an interpolated frame based on a correction difference value calculated through a region-limited motion vector calculation module (530), a region-limited representative motion vector prediction module (540), and a correction vector value calculation module (550) for a motion sustained region (811, 821, 822) identified in each of the plurality of input frames (810, 820, 830). In this case, the processor (120) can generate an interpolated frame based on a conventional motion estimation for a region (812, 822, 832) other than the motion sustained region (811, 821, 822) identified in each of the plurality of input frames (810, 820, 830). For example, the processor (120) can generate an interpolation frame based on a motion estimation value obtained through the motion estimation module (510) for an area (812, 822, 832) other than the motion continuation area (811, 821, 822).

[0139] In addition, the processor (120) may perform motion compensation for each region for a plurality of input frames (810, 820, 830) included in the input image. For example, the processor (120) may perform motion compensation for a motion sustained region (811, 821, 822) identified in each of the plurality of input frames (810, 820, 830) based on a compensation difference value calculated through a region-limited motion vector calculation module (530), a region-limited representative motion vector prediction module (540), and a compensation vector value calculation module (550). In addition, the processor (120) may not perform motion compensation for a region (812, 822, 832) other than the motion sustained region (811, 821, 822) identified in each of the plurality of input frames (810, 820, 830).

[0140] As described above, according to various embodiments, by identifying areas in an input image where motion occurs consistently and continuously, and detecting and compensating for any interruptions or jumps due to motion pauses in those areas, smooth, continuous motion can be provided in the output image. This prevents viewers from perceiving motion interruptions, thereby improving the image quality of the output image.

[0141] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0142] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0143] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0144] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0145] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0146] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, A memory storing one or more instructions; and comprising one or more processors; The one or more processors, by executing the one or more instructions, Identify areas in the input image where motion is maintained, Identifying movement information of the identified area and representative movement information of the identified area, Identifying a motion compensation value based on the identified motion information and the identified representative motion information, Motion compensation is performed on the input frame included in the input image by applying the identified motion compensation value to the identified region, Obtaining an interpolated frame by applying the identified motion compensation value to the identified area, An electronic device that obtains an output image based on an input frame on which the motion compensation is performed and the obtained interpolated frame.

2. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that obtains the interpolated frame by using the identified motion compensation value for the identified area and not using the identified motion compensation value for an area outside the identified area.

3. In paragraph 2, The one or more processors, by executing the one or more instructions, An electronic device that performs motion compensation for the input frame by using the identified motion compensation value for the identified area and not using the identified motion compensation value for an area outside the identified area.

4. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that obtains a plurality of motion vectors corresponding to the identified region from a plurality of input frames included in the input image, and identifies a representative motion vector that is continuously maintained in the identified region based on the obtained plurality of motion vectors.

5. In paragraph 4, The one or more processors, by executing the one or more instructions, An electronic device that identifies a motion compensation value corresponding to each of the plurality of input frames based on a difference value between each of the plurality of motion vectors corresponding to the identified area and the representative motion vector.

6. In paragraph 5, The one or more processors, by executing the one or more instructions, An electronic device that identifies a motion compensation value corresponding to each of the plurality of input frames by applying a preset filter to each of a plurality of motion vectors corresponding to the identified area and a difference value of the representative motion vector.

7. In paragraph 1, The one or more processors, by executing the one or more instructions, Identifying an area where the motion is maintained based on motion vector statistics for each of a plurality of pixel areas included in the input image, or An electronic device that inputs the input image to a learned artificial intelligence model to identify an area where the movement is maintained.

8. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that identifies a movement distance at which the sum of differences in luminance values ​​of pixels included in the identified area among a plurality of input frames included in the input image is minimum as a motion vector of the identified area.

9. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that identifies pixels included in the identified area into a plurality of pixel areas, and identifies a motion vector of the identified area based on statistical values ​​of the motion vector identified in units of the plurality of pixel areas.

10. In paragraph 1, The area where the above movement is maintained is, An electronic device comprising an area in which text continuously moves at a specific location in the input image.

11. In a method for controlling an electronic device, A step of identifying an area in which motion is maintained in an input image; A step of identifying movement information of the identified area and representative movement information of the identified area; A step of identifying a motion compensation value based on the identified motion information and the identified representative motion information; A step of obtaining an output image by applying the identified motion compensation value to the identified area; A step of performing motion compensation for an input frame included in the input image by applying the identified motion compensation value to the identified area; A step of obtaining an interpolated frame by applying the identified motion compensation value to the identified area; and A control method comprising: a step of obtaining an output image based on an input frame on which motion compensation has been performed and the obtained interpolated frame.

12. In paragraph 11, The step of obtaining the above output image is: A control method, comprising: a step of obtaining the interpolated frame by using the identified motion compensation value for the identified area and not using the identified motion compensation value for an area other than the identified area.

13. In paragraph 12, The step of obtaining the above output image is: A control method further comprising: a step of performing motion compensation for the input frame by using the identified motion compensation value for the identified region and not using the identified motion compensation value for a region other than the identified region.

14. In paragraph 11, The step of identifying the movement information of the identified area and the representative movement information of the identified area is as follows. A control method for obtaining a plurality of motion vectors corresponding to the identified region from a plurality of input frames included in the input image, and identifying a representative motion vector that is continuously maintained in the identified region based on the obtained plurality of motion vectors.

15. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, The above actions are, A step of identifying an area in which motion is maintained in an input image; A step of identifying movement information of the identified area and representative movement information of the identified area; A step of identifying a motion compensation value based on the identified motion information and the identified representative motion information; A step of obtaining an output image by applying the identified motion compensation value to the identified area; A step of performing motion compensation for an input frame included in the input image by applying the identified motion compensation value to the identified area; A step of obtaining an interpolated frame by applying the identified motion compensation value to the identified area; and A non-transitory computer-readable medium comprising: a step of obtaining an output image based on an input frame on which motion compensation has been performed and the obtained interpolated frame.

Images